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ABSTRACT 



The effects of an interactive multimedia program on 
teachers' understanding and implementation of an instructional innovation 
were examined. Fifty-eight preservice teachers and 10 inservice teachers 
participated and were randomly assigned to one of two teacher development 
programs. The first teacher development program, called the ’'Virtual 
Workshop, " was a computer based, interactive multimedia program. The second 
teacher development program, called the "Actual Workshop," was a traditional, 
live presenter directed program. Results of the study indicate that compared 
to their pretest scores, the posttest scores earned by inservice and 
preservice teachers on the test of knowledge and understanding of the 
innovation significantly improved following participation in either workshop. 
Moreover, preservice and inservicc' teachers ' satisfaction ratings of both 
workshops were favorable. Inservice teachers who participated in the 
workshops correctly performed a substantially greater number oj. 
innovation's targeted behaviors after training than before training. The 
development cost of the Virtual Workshop was found to be nearly four times as 
expensive as the development cost of the Actual Workshop; however, the cost 
to implement the Virtual Workshop was less than the Actual Workshop. Overall, 
both workshops had similar, positive effects on both groups of teachers' 
understanding and implementation of the instructional innovation. (Contains 
58 references.) (Author/CR) 
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Abstract 



The effects of an interactive multimedia program on teachers' understanding and 
implementation of an instructional innovation were examined. Fifty-eight preservice 
teachers and 10 inservice teachers participated and were randomly assigned to one of 
two teacher development programs. The first teacher development program, called the 
Virtual Workshop, was a computer based, interactive multimedia program. The second 
teacher development program, called the Actual Workshop, was a traditional, live, 
presenter-directed program. Results of the study indicated that compared to their pretest 
scores, the posttest scores earned by inservice and preservice teachers on the tests of 
knowledge and understanding of the innovations^-; significantly improved following 
participation in either the Virtual or Actual Workshops. Moreover, preservice and 
inservice teachers' satisfaction ratings of both the Virtual and Actual Workshops were 
favorable. Inservice teachers who participated in the Virtual and Actual Workshops 
correctly performed a substantially greater number of the innovation's targeted 
behaviors after training than before training. Overall, both the Virtual and Actual 
Workshops had similar, positive effects on both teachers' understanding and 
implementation of the instructional innovation. This study suggests that interactive 
multimedia programs like the Virtual Workshop may provide a new medium through 
which effective teacher development can be provided. 
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Chapter 1 
Introduction 

The students in America's classrooms are becoming increasingly diverse 
(Hodgkinson, 1991). This diversity is not only ethnic, cultural, and economic, but also 
academic. One factor contributing to the increase in academic diversity has been the 
movement to include students with mild disabilities within general education 
classrooms for most if not all of the school day (Gartner & Lipsky, 1987; Kauffman, 
1994; Will, 1986). Needless to say, appropriately addressing the educational needs of 
these students in general education classrooms is a challenge for teachers (Simmons 
et al., 1995; Zigmond et al., 1995). Although, teachers have indicated that they want 
to meet this challenge, they realize their need for training in effective inclusive 
practices (Joint Committee on Teacher Planning for Students with Disabilities, 1995). 

Although inclusive practices designed to improve the achievement of students 
with disabilities in inclusive classes have been developed and validated (Fisher, 
Schumaker, & Deshler, 1995), teacher development programs have not proved 
successful in translating such instructional innovations into classroom practice on a 
broad scale (Gersten, Vaughn, Deshler, & Schiller, 1995; Malouf & Schiller, 1995). 
Often, these programs involve one-shot sessions (Kline, Deshler, & Schumaker, 

1991), focus on practices frequently not perceived as needed by teachers (Fullan with 
Stiegelbauer, 1991), provide few, if any, opportunities to practice and receive 
feedback (Cruickshank & Metcalf, 1990), and offer little or no follow-up classroom 
support (Hoover & Boethel, 1991). Such episodic teacher development programs 
contrast directly with more successful comprehensive teacher development programs 
that are needs-based, participant-owned, and supported over time (Shower, Joyce, & 
Bennett, 1987; Schumaker & Clark, 1990). Still, comprehensive programs are more 
expensive than episodic programs (Korinek, Schmid, & McAdams, 1985) and may be 
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beyond the means of today's public schools. Clearly, if general education teachers are 
going to more successfully address the educational needs of all students enrolled in 
academically diverse classrooms, improving teacher development programs in ways 
that facilitate teachers' adoption of validated inclusive practices in cost-effective ways 
must be found. 

One emerging technology which eventually might be used to provide teacher 
development programs is interactive multimedia. Interactive multimedia is a term 
used to describe computer-based programs that provide users random access to 
multiple forms of media (i.e., text, graphics, audio, video, etc.) about a particular 
topic (e.g., Kinzie & Berdel, 1990; Marchionini, 1988). For example, when studying 
about semantic maps using an interactive multimedia program, a teacher might access 
text passages describing semantic maps, diagrams illustrating semantic maps, and 
video clips showing teachers using semantic maps. Moreover, the teacher would have 
control over which media to view, when to view them, and how many times to view 
each. 

The availability of interactive multimedia programs for teacher development 
could shift the balance from less effective episodic education programs to more 
effective comprehensive ones. For example, a library of CD-ROM based, interactive 
multimedia programs on innovative practices could be provided to teachers. From this 
library, teachers could choose programs on those innovative practices that would fit 
their most pressing instructional needs. While using a chosen interactive multimedia 
program, teachers could practice using the innovation in a simulated lesson and 
receive feedback on their performance. Moreover, a teacher could review the program 
as frequently as desired. Such teacher-directed training would presumably enable 
instruction to become more individualized, in a format that is potentially less 
expensive than "one shot," "one-size-fits-all" programs (Schrum, 1994). The savings 
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accrued through interactive multimedia programs could then be used to provide 
teachers follow-up support with consultants and/or other teachers. 

Additionally, interactive multimedia programs could afford school districts the 
option of bringing new teachers up to speed with other staff in a relatively short 
period of time because the new teacher would have immediate access to the training. 
They would not have to wait for the next "formal" training session. Finally, because 
CD-ROM based, interactive multimedia programs can be distributed with great ease 
through the mail, professionals would have timely access to state-of-the-art 
instructional innovations. 

These potential advantages make interactive multimedia programs an 
attractive form of training. Not surprisingly, numerous interactive multimedia 
programs for university students and professionals have been developed on a broad 
array of subjects including: accounting (Becker & Dwyer, 1994), art (Covey, 1990), 
biology (Hannaway, Shuler, Bolte, & Miller, 1992; Hutchings, Hall, & Thorogood, 
1994; Jaffe & Lynch, 1989), business (Acovelli & Nowakowski, 1994), foreign 
language (Liu & Reed, 1995), history (Chignell & Lacy, 1988; Spoehr & Spoehr, 
1994), literature (Landow, 1989), medicine (Lee, Ault, Kirk, & Comstock, 1995), and 
statistics (Egan et al, 1989; Harding, Lay, Moule, & Quinney, 1995; Johnson & 
Grover, 1993). 

The surge in interactive multimedia program development for training 
purposes suggests that many individuals believe such programs can deliver effective 
instruction to adults. In fact, several methodologically sound empirical studies 
support this belief (D'Alessandro et al., 1993; Hudson & Holland, 1992; Lanza & 
Roselli, 1991; Livergood, 1994; McGrath, 1992; Patterson & Yaffe, 1993; Quade, 
1993; Santer et al., 1995; Schank & Rowe, 1993; Shyu & Brown, 1992; Summers, 
1991; van den Berg & Watt, 1991). Combined, these studies suggest that interactive 
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multimedia programs are at least as effective as and sometimes more effective than 
traditional means of instruction. Moreover, consumer satisfaction results consistently 
indicate that students rate the interactive multimedia programs favorably. 

Nevertheless, despite the promise these programs hold, no methodologically 
sound empirical studies have examined the effects of interactive multimedia programs 
on the professional development of classroom teachers. Moreover, except for a study 
by Shyu and Brown (1992) on origami, no studies have examined the effect of 
interactive multimedia programs on adults' ability to use what they have learned on an 
authentic task. If interactive multimedia programs are going to be used for teachers’ 
professional development, studies must be conducted to determine whether such 
programs: a) are effective with regard to teaching preservice and inservice teachers 
new knowledge about innovative practices; b) are effective with regard to teaching 
teachers to implement innovations in their classrooms with students; and c) are 
efficient with regard to cost. The purpose of this study was to directly address the first 
two of these issues and to shed light on the third. 
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Chapter 2 

Review of Literature 

Over the past decade, professionals in the field of education have actively 
sought to learn how to more effectively translate research into practice on a broad 
scale (Lenz & Deshler, 1993; Malouf & Schiller, 1995). Their investigative efforts 
have substantially advanced the recognition of key factors which promote teachers' 
understanding and implementation of research-based practices (e.g., Gersten & 
Brengelman, 1996). One of these factors is comprehensive teacher development 
(Cruickshank & Metcalf, 1990; Fullan with Stiegelbauer, 1991; Guskey, 1986; 1994). 

Much is known about how to effectively provide comprehensive teacher 
development (e.g., Cruickshank & Metcalf, 1990; Showers et al., 1987; Sparks, 1983; 
Wade, 1985), and today, validated models for such teacher development exist (e.g.. 
Gall & Vojtek, 1994; Sparks & Loucks-Horsley, 1989). One such model for 
comprehensive teacher development was described by Joyce and Showers (1988). In 
this model, an instructional need is first targeted and teachers make a written 
commitment to participate fully in all aspects of a professional development program 
to address that need. Then teachers gain knowledge of and skill in an instructional 
innovation designed to meet the targeted need in a workshop. During the workshop, 
the theory underlying the innovation is described, demonstrations of the innovation 
are presented, and opportunities to practice and receive feedback using the innovation 
are provided. Following the workshop, teachers then participate as members of peer- 
coaching teams supporting one another's use of the innovation. The effectiveness of 
this model was tested in a study that measured teachers' ability to integrate four 
innovations (cooperative learning structures, inductive thinking, concept attainment, 
and mnemonic strategies) into their classroom practice (Showers, 1990). Results of 
this study indicated that following the initial workshop, participating teachers used 
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these innovations an average of 20 times each month. Moreover, after nearly two 
years, 60% of the participating teachers continued to implement these innovations 
effectively. 

A second comprehensive teacher development model, comprised of four 
phases, was described by Schumaker and Clark (1990). In phase one of this model, 
teachers, administrators, and staff developers work to identify an area of need and 
instructional innovations that might address that need. In phase two, teachers’ 
commitment to learn and implement the innovation is obtained. Then, in a workshop, 
the philosophy behind the innovation and the innovation itself is described, a 
demonstration of the innovation is presented, opportunities to practice and receive 
feedback using the innovation are provided, and all the materials required to 
implement the innovation are distributed. Following the workshop, teachers meet as 
members of support teams to discuss implementation issues. After the initial 
innovation has been implemented successfully, phase three of the model is then 
initiated. During this phase, additional innovations are learned and integrated into a 
coordinated instructional program. Finally, in phase four, district policies are 
developed to facilitate the maintenance of this coordinated instructional program. 

In a recent series of studies, aspects of this teacher development model were 
tested with regard to teachers' implementation of learning strategies instruction (Kline 
et al., 1991). Results of one study indicated that when this model was applied, all 
participating teachers were able to initially implement the learning strategies 
instruction, and 75% percent of the teachers taught the strategies throughout the 
school year. When aspects of the model such as providing materials, feedback, or 
support-team meetings, were omitted systematically, at best only half of the 
participating teachers implemented the learning strategies instruction. Moreover, 
these teachers implemented the instruction less efficiently and with significantly 



fewer students than teachers who participated in all components of the teacher 
development model. 

Stallings (1989) described a third comprehensive teacher development model. 
In this model, teachers first work to identify an instructional need and then make a 
written commitment to learn and implement a new practice which address that need. 
Next, teachers attend small-group workshops and learn about the practice through a 
variety of means including explanations, models, and practice simulations. Teachers 
are then encouraged to integrate the practice into their classrooms and to evaluate the 
effects of the new practice on student learning. Once they have used the practice, 
teachers are observed and their performance is critiqued. Finally, teachers participate 
as members of support groups in which teachers report their success using the new 
practice, discuss problems they have experienced, develop solutions to these 
problems, and set new goals for professional growth. 

The effectiveness of this model was examined and reported in two studies in 
which teachers' knowledge and implementation of a new reading program were 
measured (Stallings, 1989). Results of the first study indicated that after participating 
in the professional development model, all 47 teachers successfully implemented the 
reading program with regard to 26 of 31 variables. Results of the second study 
indicated that after participating in the model, at least 80% of the teachers in three of 
four participating schools implemented the program successfully. Moreover, the 
teachers' implementation of the reading program resulted in substantially higher 
student reading scores. That is, students of teachers who received training could read, 
on the average, half a grade level higher than students of teachers who did not receive 
training. 

Each of these three comprehensive teacher development models has proved to 
be effective. Overall, these models were predicated on developing teachers' 



knowledge of instructional innovations and their skill in applying those innovations 
(Cruickshank & Metcalf, 1990). To develop teachers' knowledge and skills, these 
models employed many similar instructional principles which included: (a) having 
teachers identify a pressing instructional need; (b) having teachers make a 
commitment to adopt the new instructional innovation; (c) explaining the theory or 
rationale behind the innovation; (d) describing the innovation; (e) providing actual 
models and/or demonstrations of the innovation; (f) providing teachers opportunities 
to practice using the innovation; (g) providing teachers feedback about their use of the 
innovation; and (h) providing follow-up support. 

Unfortunately, despite the effectiveness of comprehensive teacher 
development programs for helping teachers to understand and implement new 
practices, such programs are rarely applied (Hoover & Boethel, 1991). The reasons 
for such limited application are largely speculative. Possible reasons include the 
difficulty and the expense of arranging for qualified experts to lead a series of 
professional development sessions with teachers while covering instruction for 
students in these teachers' classes (Korinek, Schmid, & McAdams, 1985). Clearly, if 
the translation of research into classroom practice is going to occur on a broad scale, 
comprehensive teacher development programs that are widely embraced must be 
found. 

One possible medium through which an embracable comprehensive teacher 
development program could be delivered is educational technology. Recent advances 
in computer and communications technologies have dramatically expanded teachers' 
opportunities to receive and to participate in professional development programs in 
exciting ways (Thornburg, 1992). Today, for example, first- year teachers attending 
Harvard University can participate in an on-line mentorship program with master 
teachers (Merseth, 1992); students in the School of Education at the University of 



Oregon can enroll in courses offered over the Internet (Schrum, 1992); and teachers 
from rural school districts all over the nation can partake in workshops delivered over 
interactive telecommunication networks (Slaton & Lacefield, 1991). 

Another recently developed computer-based technology now being used to 
provide professional development programs to teachers is interactive multimedia 
(Kinzie & Berdel, 1990; Marchionini, 1988). Interactive multimedia has great 
potential for developing teachers' knowledge of and skill in applying an instructional 
innovation, for such programs can contain many of the instructional principles present 
in comprehensive professional development. For example, when learning about an 
instructional innovation using an interactive multimedia program, teachers could view 
a video clip of an individual explaining the theory or rationale behind the innovation. 
Additionally, the innovation could be described to teachers in great detail, and they 
could view video clips of demonstration lessons performed by teachers using the 
innovation with students. To check their own understanding of the innovation, 
teachers could answer questions integrated into the program and receive immediate 
positive and comctiwe feedback regarding their answers. Moreover, teachers could 
practice making decisions about how to use the innovation in a scenario-based 
simulation. That is, teachers could read a scenario describing a lesson in which the 
innovation was to be used with students. At points in the scenario, teachers could 
respond to questions requiring them to determine how the innovation should be best 
used. Teachers could be giwen feedback on these responses. The flexibility of 
interactive multimedia programs would also allow teachers to determine the order and 
pace at which they learn about the innovation. Additionally, teachers could review 
any section of the program, at any time, as often as needed. 

In recent years, numerous interactive multimedia programs have been 
developed for university students and professionals in a wide range of disciplines 
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(Acovelli & Nowakowski, 1994; Becker & Dwyer, 1994; Chignell & Lacy, 1988; 
Covey, 1990; Egan et al., 1989; Hannaway et al., 1992; Harding et al., 1995; 
Hutchings et al., 1994; Jaffe & Lynch, 1989; Johnson & Grover, 1993; Landow, 

1989; Lee et al., 1995; Liu & Reed, 1995; Spoehr & Spoehr, 1994). Moreover, 
interactive hypermedia programs have also been developed for preservice and 
inservice teachers on topics such as classroom management (Overbaugh, 1994), 
mainstreaming (Rojewski, Gilbert, & Hoy, 1994), and measuring the behavior states 
of students with multiple disabilities (Bashinski, 1996). 

Clearly, this surge in the development of interactive multimedia programs 
suggests many individuals believe such programs can effectively provide instruction 
to adults. However, the mere popularity of hypermedia programs is not enough to 
support their continued development and use. If interactive multimedia programs are 
to be used in the professional development of preservice and inservice teachers, the 
effects such programs have on adults' ability to understand and apply what is being 
taught must be determined. The purpose of this review is to describe and critically 
examine the research on existing interactive multimedia programs in light of the 
instructional principles found in effective teacher development programs. 

Selection of Studies 

The research studies included in this review were identified by searching the 
CD-ROM databases for the Educational Resources Information Center, Psychological 
Abstracts, and MedLine for the years 1980-1996. In addition, an ancestral search 
from the identified articles was conducted. To be selected for the review, a study had 
to meet the following criteria: 

a) The study had to be conducted with adults. Adults were defined as 
individuals over the age of 18 and/or attending a post- secondary education 
program (i.e., university, community college, or professional workshop). 
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(Since few studies have been conducted on the effects of multimedia programs 
with teachers, the review was broadened to include any studies on adults.) 

b) The study had to employ as an independent variable an interactive 
multimedia program. For this review, "interactive multimedia" programs were 
defined as learner-directed, computer-based applications, in which text, 
graphics, video, and/or audio segments were integrated. 

c) The study had to report empirical data on the effects the interactive 
multimedia program on adults' knowledge and/or skill development. 

d) The study had to employ an experimental design and control for the effects 
of extraneous variables. 

Once selected, the studies were sorted into two categories of interactive 
multimedia: hypertext and hypermedia. Each study was then reviewed to identify the 
number of adults participating, the characteristics of those adults, the setting in which 
the study was conducted, the adequacy of the research design used, the dependent 
variables being measured, the outcomes for the participating adults, and the level of 
the participating adults' satisfaction with the interactive-multimedia program. 

Each category of interactive multimedia and the studies related to it will be 
described. Moreover, the effectiveness of the interactive multimedia programs will be 
summarized with respect to how they affected adults' knowledge and/or skill 
development. Finally, conclusions will be drawn with regard to how well these 
interactive hypermedia programs match the principles of effective teacher 
development described above. 

Hypertext Proerams 

Hypertext comprises one category of interactive multimedia. Originally 
conceived in the 1940's by Bush (Quade, 1993), hypertext provides user-driven access 
to static forms of media (i.e., text, maps, photographs, and graphic displays) related to 



a particular topic. These media are connected together by hypertext links, such as 
colored words imbedded within a passage of text. Using a mouse, these words can be 
"clicked." When clicked, additional information about that word appears on the 
computer screen. For example, if reading a passage about individualized education 
plans (lEPs) within a hypertext program, a teacher might click on any word from that 
passage which appears in blue, boldfaced print. If the teacher clicks the blue, 
boldfaced words, annual goal, sample annual goals from an actual lEP could appear 
on the screen for the teacher to study. If the teacher clicks the blue, boldfaced word, 
committee, the names and descriptions of those individuals required to attend an lEP 
meeting could also appear on the screen. Moreover, if the teacher choose to reread the 
passage, any previously clicked words could again be accessed for review. 

Six studies examining the effects of hypertext on adults' learning have been 
included in this review. In one study, Livergood (1994) compared the effects of two 
hypertext programs and an instructional manual on 209 university students’ 
knowledge of intelligent computer systems. The first hypertext program was 
comprised of three instructional modules. Modules one and two contained 
information and examples describing intelligent computer systems. Module three 
contained review questions students could answer about intelligent computer systems. 
Whether students received corrective feedback on questions they did not answer 
correctly is unclear. The second hypertext program was exactly like the first; 
however, it did not contain the third instructional module. The content contained 
within the instructional manual was not disclosed. Participating students were 
randomly assigned to one of the three instructional treatments, were given up to 45 
minutes to complete their respective instructional treatment, and then completed a 20- 
item posttest. The nature of this posttest was not described. 
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The posttest scores earned by students in the three-module hypertext program 
were significantly higher than the scores earned by students in the other two 
instructional treatments. Specifically, students who participated in the three-module 
hypertext program earned an average posttest score of 91.71%. Students who received 
the two-module hypertext program earned an average posttest score of 84.37%. 
Students who used the instructional manual earned an average posttest score of 
85.68%. Unfortunately, because a pretest was not given, the magnitude of the 
students' growth was not determinable. 

In a similar study, Patterson and Yaffe (1993) measured and compared the 
effects of a hypertext program and an instructional manual on 22 social-work 
students’ skill in diagnosing personality and developmental disorders. These students 
were randomly assigned to one of two instructional treatments. The hypertext 
program, called HyperAxis II, contained information describing characteristics, 
diagnostic information, and case studies of personality and developmental disorders. 
Moreover, the program contained written scenarios describing a person with an 
undisclosed personality or developmental disorder. From the information provided in 
the scenarios, students could practice making diagnoses and receive corrective 
feedback when errors occurred. The instructional manual contained the same 
information found in HyperAxis II. The instructional manual even contained the 
practice scenarios. However, to check the accuracy of their diagnoses, students had to 
look at an answer key which provided the correct answer but did not provide 
corrective feedback. Students in both treatments had a maximum of three hours to 
complete their respective treatment. 

Using a pretest-posttest control-group design, participating students' skill in 
accurately diagnosing personality and developmental disorders was measured and 
compared. Results of the study revealed no significant difference between the average 



posttest score of 67.6% earned by students who participated in the hypermedia 
program and the average posttest score of 71.7% earned by students who used the 
instructional manual. Moreover, when compared to their pretest scores, neither 
group's posttest scores were significantly higher. The interscorer reliability of the tests 
was low, however. For example, the kappa coefficient for the pretest was .55, whereas 
the kappa coefficient for the posttest was .61. Thus, the accuracy of these knowledge 
scores is uncertain. Nevertheless, students in both treatments spent significantly less 
time making diagnoses on the posttest when compared to the pretest. Specifically, 
diagnosis time for students in the HyperAxis II program for six diagnoses fell from a 
pretest mean of 27.1 minutes to a posttest mean of 17.4 minutes and for students who 
used the instructional manual from a pretest mean of 26.5 minutes to a posttest mean 
of 16.4 minutes. Regarding students' satisfaction ratings of the two treatments, 
students who used the training manual rated their program as easier to use than 
students using the hypertext program. On the other hand, students who used the 
HyperAxis II program reported their program as more enjoyable than students who 
used the training manual. 

The effects of hypertext programs on university students' knowledge were also 
measured and compared to the effects of computer-based tutorial programs in two 
recent studies conducted by Quade (1993) and Lanza and Roselli (1991). In the study 
conducted by Quade (1993), the hypertext program contained nine sections of 
information, including examples and definitions describing intellectual property law. 
Embedded within these sections were questions which students could answer using a 
mouse or the computer keyboard. When questions were answered incorrectly, 
corrective feedback was provided. The computer-based tutorial was comprised of the 
same content and questions that were provided in the hypertext program; however, 
the content was presented in a predetermined order. 
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Seventy-six undergraduate students enrolled in a business law course were 
randomly assigned to one of the two instructional treatments. Before and after 
receiving their respective instructional treatment, these students completed a pretest 
and posttest which contained the same 30 multiple-choice questions. Results of the 
study revealed no significant differences between the pretest scores and between the 
posttest scores earned by students in the two treatment groups. However, compared to 
their pretest scores, the posttest scores of students within each treatment were 
significantly higher. More specifically, students who participated in the hypertext 
program, on average, earned a pretest score of 43.50% and a posttest score of 62.73%. 
Students who participated in the computer-based tutorial, on average, earned a pretest 
score of 41.37% and a posttest score of 61.80%. Still, given that students were 
pretested and posttested using the same multiple-choice questions, the magnitude of 
these students' improvement is modest, at best. 

In the study conducted by Lanza and Roselli (1991), the hypertext program 
described the Pascal computer language and provided students opportunities to 
answer multiple-choice and fill-in-the-blank questions about the language. After 
answering a question, students received feedback indicating whether their response 
was correct or incorrect. Moreover, incorrect answers were followed by another 
question or corrective material. The computer-based tutorial contained the same 
content and questions as the hypertext program; however, the tutorial was computer- 
directed, not user-directed like the hypermedia program. That is, the computer, not the 
user was in control of what content was to be presented and when it was to be 
presented. Additionally, the feedback provided to students in the tutorial merely 
indicated the correctness or incorrectness of an answer. No corrective feedback was 
given. 
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Sixty undergraduate students studying computer science were randomly 
assigned to either the hypertext program or the tutorial. Students were provided the 
same amount of time to complete their respective instructional treatment, and they 
completed the same 10-item posttest. The nature of the posttest questions was not 
disclosed, nor was the interscorer reliability for the test scores. On the posttest, 
students who participated in the hypertext program earned an average score of 
49.67%; students who participated in the computer-based tutorial earned an average 
score of 53.33%. The difference between the scores earned by students in the two 
treatments was not statistically significant. Although the groups of students performed 
similarly, 76% of the students who participated in the hypertext program reported 
finding their program stimulating and attractive, whereas only 37% of the students 
rated the tutorial this way. 

In a comprehensive study, McGrath (1992) examined the effects of a 
hypertext program, a computer-based tutorial, and an instructional manual on both 
university students' knowledge of and skill in calculating surface area. One hundred 
and three undergraduate education majors participated in the study. These participants 
were randomly assigned to one of the three instructional treatments. All three 
treatments contained identical content and practice problems on surface area. 
Moreover, the content and practice problems were organized into the same sections. 
The computer-based tutorial differed from the hypermedia program in that its sections 
had to be completed in a predetermined order. The instructional manual differed from 
the hypertext program and the computer-based tutorial in that it was in a bound, paper 
format. 

To measure and compare the effects of each treatment, a pretest-posttest, 
control-group design was used. Overall, results of the study revealed no significant 
differences between the posttest knowledge scores earned by students in any of the 
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three treatments. Significant differences were found with regard to the skill measure; 
however, because no post-hoc analyses were conducted, specifying whether the 
students who used the instructional manual performed significantly better than the 
students who participated in the hypertext program or the students who participated in 
the computer-based tutorial is not possible. Interestingly though, the students who 
used the instructional manual spent significantly more time than other groups 
completing the instruction. No significant differences between the students' 
satisfaction ratings were found; students participating in all three treatments were 
satisfied with the instruction they received. 

Finally, van den Berg and Watt (1991) studied the effects of a hypertext 
program on university students' knowledge of statistics. This study was unique, 
because it did not compare a hypertext program to another instructional treatment', 
instead, it compared the effects of the same hypertext program under varying 
instructional conditions. The hypertext program contained hierarchically organized 
content about hypothesis testing. When abstract information was presented, the user 
could access less abstract definitions or examples to make the content more concrete. 
The four instructional conditions were as follows. In the first condition, called the 
replacement condition, students used the hypertext program in place of class lecture 
for a period of six weeks. In the second condition, called the supplementary 
condition, students used the hypertext program to supplement class lecture. In the 
third condition, called the competitive condition, students used the program in place 
of class lecture. Moreover, these students were told they were receiving their 
instruction in a format different from that received by their classmates. In the fourth 
condition, called the control condition, students did not use the hypertext program and 
simply attended class lectures. The content presented in the hypertext program was 
identical to that presented in class lectures. 
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For the study, 241 students were randomly assigned to one of these four 
instructional conditions. Initially, all the participating students first attended five 
weeks of class lecmre and completed an exam over the content presented. Analysis of 
these exam scores revealed no significant differences between the groups. The four 
instructional conditions were then implemented. Afterward, students completed a 
second exam over the content presented in the four conditions. Results indicated that 
the scores earned by students in the control condition were significantly higher on this 
second exam than those earned by students in the replacement condition. No other 
significant differences between instructional conditions were found. Following the 
second exam, all participating students again attended five weeks of class lecture and 
completed a third exam. Analyses revealed no significant differences between the 
scores earned by students assigned to any of the four conditions. However, because 
actual scores were not reported and the level of student achievement was not revealed, 
the meaningfulness of student performance on each of these exams in unclear. In 
terms of consumer satisfaction, students in the supplementary condition rated the 
hypertext program the most favorably; students in the competitive condition rated 
their hypertext program the least favorably. 

Hypermedia Programs 

Hypermedia is a second type of interactive multimedia. Like hypertext, 
hypermedia is a computer-based program that allows user-directed access to multiple 
forms of static media. Additionally, however, hypermedia also allows access to 
dynamic media like audio and video segments. For example, if reading a passage 
about individualized education plans (lEP) within a hypermedia program, a teacher 
might click on the blue, boldfaced word, negotiation, and a video segment of a 
teacher and parent working to determine a child's course schedule might appear on the 
screen for the teacher to view. The teacher could also click the blue, boldfaced word. 
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invitation, to hear an audio segment of a teacher asking parents to attend their child's 
lEP conference. Moreover, like in hypertext, in hypermedia any previously clicked 
words could again be accessed for review. 

Six studies examining the effects of hypermedia on adults have met the 
criteria for this review. In the first study. Summers (1991) compared the effects of a 
hypermedia program and an instructional manual on university students' knowledge of 
pharmaceuticals. Twenty-one students participated in the study and were randomly 
assigned to one of the two treatments. The instructional manual contained content 
divided into seven sections. To provide additional information to what was presented 
in the manual, segments from a videotape on pharmaceuticals were shown. Beginning 
with section one, students read the manual, viewed excerpts from the videotape at 
when instructed by the manual, answered questions in the instructional manual, and 
checked answers using an answer key. This process was continued for the subsequent 
sections of the instructional manual. The hypermedia program was comprised of the 
same content and video segments as the instructional manual. This information was 
transformed into a computerized format that could be read and viewed in any order. 
Also, students could stop, reverse, and replay the video excerpts. When students 
answered questions incorrectly they received corrective feedback. 

Two weeks after students completed their respective treatments, they 
completed a posttest. The mean posttest score of students who participated in the 
hypermedia program was not statistically different from the posttest score of those 
using the manual. Likewise, the amount of time students needed to complete each 
instructional treatment did not differ significantly. Still, students who participated in 
the hypermedia program rated their enjoyment of the hypermedia program 
significantly higher than students who used the instructional manual. 
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In two later studies, Hudson and Holland (1992) and D' Alessandro et al. 

(1993) measured and compared the effects of a hypermedia program and large-group 
lecture on university students' knowledge of course content. In the study conducted by 
Hudson and Holland (1992), the hypermedia program contained text and graphics on 
lighting techniques taken from the students' textbook and video excerpts from a 
videotape entitled, "Lighting in the Real World." The large-group lecture included the 
same content as the hypermedia program; however, the nature of the content was not 
described. Thus, whether models of effective lighting techniques and/or practice 
activities were provided remains unclear. During the lecture, the video excerpts were 
shown on a television monitor, the graphics were presented on the chalkboard, and 
the content was referred to in their textbook. Whether either treatment included 
practice activities and/or feedback was not indicated. Students in both groups received 
instruction on lighting techniques for no more than fifty minutes, the length of one 
class period. 

Eighty-eight university students were randomly assigned to one of the two 
treatments, and a pretest-posttest control-group design was used. Results of the study 
revealed no statistically significant difference between the posttest knowledge scores 
of students in the two treatment conditions. However, compared to pretest scores, the 
posttest scores of students within each treatment were significantly higher. Later, 
when asked to rate their enjoyment of the instruction, students who participated in the 
large-group lecture indicated they enjoyed their lesson about the same as previous 
lessons. However, the students who participated in the hypermedia program indicated 
that they enjoyed their lesson more than previous lessons. 

In the study conducted by D'Alessandro et al. (1993), the hypermedia program 
was divided into 10 sections containing passages of text describing respiratory 
diseases. Words within these passages could be clicked to access either additional 
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text, images, video segments, audio segments, or animation. Moreover, this program 
contained practice test questions. Whether students received feedback after answering 
these questions was not explained. The large-group lecture contained content identical 
to the hypermedia program; however, no evidence is presented corroborating this 
claim. The lecture was delivered within 50 minutes. 

Forty-nine staff physicians and residents in a department of radiology were 
randomly assigned to one of the two instructional treatments. A pretest-posttest 
control-group design was used. The pretest and the posttest were comprised of the 
same 10 questions. The type of questions used is unclear. Results indicated no 
statistical difference between the two groups at either pretest or posttest. However, 
compared to pretest scores, the posttest scores of students within each treatment group 
were significantly higher. For example, professionals who participated in the 
hypermedia program, on average, earned a pretest score of 58.8% and a posttest score 
of 83.2%. Professionals who attended the group lecture, on average, earned a pretest 
score of 53.3% and a posttest score of 82.9%. Interestingly, when considering 
instructional time, the professionals who attended the lecture learned the same 
amount of information as professionals who participated in the hypermedia program 
in 40% less time. Thus, the lecture was the more efficient way of teaching. In terms of 
consumer satisfaction, 75% of the physicians and residents rated the hypermedia 
program favorably, and all indicated it was easy to use. Satisfaction ratings for those 
professionals who attended the lecture were not reported. 

In a later study, Santer et al. (1995) compared the effects of a hypermedia 
program to an instructional manual and a lecture on medical students' knowledge of 
respiratory diseases. In this study, two hundred sixty-seven, third- and fourth-year 
medical students were assigned to one of four instructional treatments. The groups of 
students did not significantly differ by age, sex, academic year, pediatric experience. 



respiratory disease knowledge, or computer proficiency. Students in the first group 
received a hypermedia program comprised of six discrete chapters containing text, 
video segments, audio segments, and diagrams. Students in the second group received 
a slide-based lecture. Students in the third group read a 37-page instructional manual. 
Except for the audio segments, the lecture and instructional manual treatments 
contained the same content as the hypermedia program. That is, all images, tables, 
and diagrams in the hypermedia program were directly abstracted and included in the 
lecture and text treatments, as were representative still images taken from the 
hypermedia program's video segments. Each treatment was completed in 60 minutes. 
Students in the fourth group served as a control group and received no instruction on 
pediatric airway diseases. This group was the only group that received a pretest on 
their knowledge of respiratory diseases. 

Following the treatment, all students were posttested on 26 multiple-choice 
questions. The posttest scores of the students who participated in the hypermedia 
program, the lecture, and the instructional manual were significantly higher than the 
posttest scores earned by control-group students. The posttest scores of students who 
participated in the hypermedia program were significantly higher than the scores of 
students who attended the lecture but were not significantly higher than those earned 
by the students who used the instructional manuak Specifically, students who 
participated in the hypermedia program, on average, earned a posttest score of 
63.84%. Students who attended the lecture, on average, earned a posttest score of 
59.23%. Students who used the instructional manual, on average, earned a posttest 
score of 60.00%. Interestingly, the mean pretest score earned by control group 
students was 45%; their mean posttest score was 51.15%. These data suggest that the 
magnitude of gain attained by students in all three treatment groups was modest. 
Regarding consumer satisfaction, 81% of the students who participated in the 
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hypermedia program rated it favorably, whereas fewer than 67% of the students in the 
other groups rated their instructional treatment favorably. 

Schank and Rowe (1993) compared the effects of a hypermedia program and a 
computer-based tutorial on university students' knowledge of semiconductor 
manufacturing. Twenty university students were randomly assigned to receive one of 
the instructional treatments. Both treatments contained five sections of content 
describing and modeling varying aspects of the semiconductor manufacturing 
process. Moreover, to illustrate dynamic aspects of this process, video segments were 
incorporated within each section. Though both treatments contained identical content 
and video segments, they were structured differently. The hypermedia program 
allowed users to select the order in which they studied the five sections, whereas the 
tutorial did not. Students had a maximum of 50 minutes to complete their respective 
treatment. 

The pretest and posttest measures given to students in both treatments were 
identical and were comprised of short-answer questions about semiconductors. 

Results revealed no significant differences between the pretest scores and between the 
posttest scores earned by the two groups of students. Although no within-group 
analyses were conducted, students in both programs did make modest gains in 
performance. That is, the average scores of students who participated in the 
hypermedia program improved from 23.20% at pretest to 40.53% at posttest. 
Similarly, the average scores of students who participated in the computer-based 
tutorial improved from 23.86% at pretest to 42.53% at posttest. 

Finally, Shyu and Brown (1992) examined the effects of a hypermedia 
program on university students' skill in constructing an origami crane. This study was 
unique, for it was the only one to measure the effects of hypermedia on students' 
ability to perform a dynamic skill. In the first treatment, a computer-based tutorial. 
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the content was presented to students in a predetermined order. More specifically, 
students were first presented an introduction, then a video summary of the paper 
folding procedure, and finally a detailed 12-step video model of the paper folding 
procedure. Finally, after each step was modeled, students had the option to view the 
video model again. However, once students proceeded to the next step, they could not 
view previously modeled steps. In the second treatment, a hypermedia program, 
students could proceeded through the content in the order of their choice. Moreover, 
students could review any content they wanted as often as they wanted. 

Participating in the study were 52 undergraduate students enrolled in an 
educational psychology course. These students had indicated they had no prior 
experience with origami, were randomly assigned to receive either the computer- 
based tutorial or the hypermedia program, and were given 30 minutes to complete 
their respective treatment. A posttest-only control-group design was employed, and 
data were analyzed using t-tests. For the posttest, students were provided a piece of 
paper which they were to fold into a paper crane. Students' cranes were rated on a 
scale ranging from 0 points to 6 points. Results of the study indicated that the cranes 
constructed by students who participated in the hypermedia program earned 
significantly higher posttest ratings than those of students who participated in the 
computer-based tutorial. Specifically, students who participated in the hypermedia 
program, on average, earned a posttest rating of 4.23, whereas students who 
participated in the tutorial, on average, earned a posttest rating of 3.13. In terms of 
consumer satisfaction, students rated both instructional treatments favorably. 

Summary 

The purpose of this review has been to describe and critically examine the 
effects of interactive multimedia programs on adults' knowledge and skill 
development and to describe the principles of effective teacher development that 
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these programs employed. Overall, this review has illustrated that few studies 
measuring and comparing the effectiveness of interactive multimedia on adults' 
professional development have been conducted. Still, much of what has been learned 
and described in the literature lends support for the use of interactive multimedia in 
professional development. For example, in all but one study (van den Berg & Watt, 
1991) the results indicated that interactive multimedia programs were at least as 
effective as instructional manuals, lecture, and computer-based tutorials on adults' 
knowledge and skill development and at times proved to be even more effective 
(Livergood, 1994; Santer et al., 1995; Shyu & Brown, 1992). Another encouraging 
finding is that all four studies that analyzed the pretest and posttest scores earned by 
adults participating in interactive multimedia programs revealed significant 
improvements in their knowledge development (D' Alessandro et al., 1992; Hudson & 
Holland, 1992; Quade, 1993) or skill development (Patterson & Yaffe, 1993). Finally, 
in all nine studies reporting consumer satisfaction, interactive multimedia programs 
were rated favorably by adults. 

Despite these promising findings, much about interactive multimedia 
programs remains unknown. First, much is unknown regarding the effects of 
interactive multimedia programs on the knowledge and skills of practicing 
professionals. No studies with inservice teachers have been conducted, and only one 
study has been conducted with preservice teachers. Moreover, the research on the 
effectiveness of interactive multimedia with adults has, in all but one case, been 
conducted exclusively with university students, individuals who are relative novices 
in their fields. Thus, because practicing professionals will likely be a substantially 
different population from the novice population, gaining a better understanding of 
how such programs affect these individuals is critical. 
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Second, the large majority of studies on interactive hypermedia have focused 
on adults' knowledge development even though this is only one dimension of 
professional development (Cruickshank & Metcalf, 1990). Knowledge is only of use 
if it can be readily accessed and applied to both related and novel situations. 
Unfortunately, few studies have examined adults' ability to apply what has been 
learned after using an interactive multimedia program, and these studies have 
primarily focused on adults' ability to apply what they have learned to paper-pencil 
problems (Patterson & Yaffe, 1993; McGrath, 1992). Only the study by Shyu and 
Brown (1992) has examined adults' ability to apply a skill (making an origami crane) 
in an authentic situation. However, because interactive multimedia programs are 
likely to be used to develop adults' skills in more sophisticated areas than folding 
paper cranes, more research is needed. 

Third, nearly all of the studies provided limited descriptions of the interactive 
multimedia programs. As a result, identifying the principles of effective professional 
development each interactive multimedia program employed was difficult. When 
descriptions were provided, the principles most commonly described as part of these 
programs were practice activities and models. Concluding with confidence whether 
the presence of any one principle or combination of principles positively affected 
adults' knowledge and skill development is difficult. What is clear from the limited 
descriptions provided is that an effort has not been made to construct interactive 
multimedia programs based on known principles of effective professional 
development. The fact that so few principles were clearly employed may help explain 
why the magnitude of gain most participants in interactive multimedia programs 

) 

achieved from pretest to posttest on knowledge measures was not as large as one 
would have hoped. 
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The magnitude of improvement from pretest to posttest on knowledge 
measures for most participating adults ranged from 17.23% to 24.40% in the studies 
described. This magnitude of improvement was often not enough to help the adults 
earn satisfactory scores (i.e., scores above 80%) on posttest knowledge measures. In 
only two studies did participating adults earn satisfactory posttest scores (i.e., scores 
above 80%). However, in one of these studies (Livergood, 1994), the magnitude of 
participants' improvement was impossible to ascertain because no pretest scores were 
reported. In the other (D'Alessandro et al., 1993), the magnitude of participants' 
improvement was only modest. The physicians who participated entered the study 
with a high level of prior knowledge of respiratory airway diseases; as a result, only 
modest improvement was required to earn satisfactory scores on the posttest. 

If the power of interactive multimedia programs for professional development 
is ever going to be fully understood, new comprehensive multimedia programs 
integrating known principles of effective professional development must be 
constructed. The effects of such programs on adults' knowledge and skill 
development must also be measured and compared to those of other instructional 
treatments. Moreover, comparisons cannot be made to "strawman" treatments. To 
make meaningful comparisons, comprehensive interactive multimedia programs must 
be compared to comprehensive instructional programs. 

Finally, none of the studies described in this review reported the costs 
associated with interactive multimedia programs and compared those costs to the 
costs of other programs. This is a significant limitation, for the likelihood of 
interactive multimedia programs being embraced will surely decrease if such 
programs are too expensive to produce and use. Future research studies need to 
describe the costs related to producing interactive multimedia programs. Once this 
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information is available, cost benefit analyses can be conducted, and the efficiency of 
such programs for professional development can be illuminated. 

If the translation of research into practice is going to occur on a broad scale, 
ways to promote cost-effective, comprehensive teacher development programs are 
needed. Clearly, the studies described in this review indicate interactive multimedia 
programs may help to fill this need. However, until the limitations of research studies 
on interactive hypermedia programs described above are addressed, determining 
whether such programs can provide teacher development effectively and efficiently 
will be difficult at best. The purpose of the following study was to address some of 
these limitations. For this study, two comprehensive teacher development programs 
were created for an innovative teaching practice. The effects of these two programs 
on preservice and inservice teachers' knowledge of and skill in implementing the 
practice were measured and compared, and the costs for developing the hypermedia 
program were outlined. The first program, called the Virtual Workshop, was a 
computer-based hypermedia program. The second program, call the Actual 
Workshop, was a traditional, live, presenter-directed program. 
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Chapter 3 
Methods 

Participants 

Preservice Teachers. Fifty-eight university students volunteered to participate 
in the study. These students were recruited from a group of 59 students enrolled in an 
instructional methods course. Of these students, 39 were undergraduates and 19 were 
graduates; 22 were earning elementary teaching certificates, 34 were earning 
secondary teaching certificates, and 2 were seeking no certification; 44 were female 
and 14 were male; and 53 were white, 3 were Asian, and 2 were Hispanic. The 
students' ages ranged from 20 to 42 years (M = 23.86, SD = 4.33). For participating, 
each student received twenty-five dollars and a copy of the Concept Mastery Routine 
teacher's manual (Bulgren, Schumaker, & Deshler, 1993). 

Inservice Teachers. Ten inservice teachers volunteered to participate in the 
study. Of these teachers, eight were general educators teachers (Grades 7-12), and 
two were special educators (Grade 7). These teachers were recruited from five 
schools in a suburban school district in eastern Kansas. Of these teachers, all held 
Bachelor's degrees and three held Master's degrees. Seven were female, and 3 were 
male. All were white. These teachers' ages ranged from 23 to 57 years (M= 35.80, SD 
= 11.18), and their years of teaching experience ranged from .50 to 24 (M= 8.80, SD 
= 9.77). For participating, each inservice teacher received fifty dollars and a copy of 
the Concept Mastery Routine teacher's manual (Bulgren et al., 1993). 

Setting 

Virtual Workshop. The Virtual Workshop took place in a classroom at a large 
midwestem university. The classroom was outfitted with 10 seats arranged in a half 
circle facing a large-screen television monitor that was connected to a computer. This 
classroom was adjacent to a Macintosh^^ computer lab containing three Centris^^ 
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660s, three Quadra^^^ 660s, two PowerMac 6100s, two PowerMac"^^ 7100s, 
and one PowerMac^^ 8100. 



Actual Workshop. The Actual Workshop took place at the same university in 
a similar classroom. This classroom was outfitted with 10 seats arranged in a half 
circle facing an overhead projector, a projector screen, and a large-screen television 
monitor connected to a videotape player. 

The Concept Mastery Routine 

The Concept Mastery Routine (Bulgren et al., 1993) is a set of instructional 
procedures designed to help teachers teach academically diverse classes of students to 
understand and master information related to key concepts (e.g., democracy, 
mammal, rhombus) found in the curriculum. The routine is comprised of a three- 
phase instructional sequence. In phase one, called "Cue," teachers provide students an 
advance organizer during which they explain that a concept is going to be learned, 
how it is going to be learned, and how students are to participate. In phase two, called 
"Do," students learn about the concept by completing a Concept Diagram (see Figure 
1) in partnership with the teacher. A Concept Diagram is a two-dimensional graphic 
device comprised of seven sections. Each section is completed with specific 
information about the concept. For example, sections are filled with: (a) 
characteristics the concept always, sometimes, and never possesses; (b) examples and 
nonexamples of the concept, and (c) a definition for the concept. Teachers and 
students complete the Concept Diagram following seven Linking Steps. The Linking 
Steps are procedures a teacher follows to ensure the Concept Diagram is completed 
accurately and in partnership with students. Once the Concept Diagram has been 
completed, phase three begins. In this phase, called "Review," teachers ask students 
questions to check their understanding of the concept and the process followed to 
analyze the concept. 
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Teacher Development Programs 



Virtual Workshop. Two-teacher development programs were prepared to 
teach inservice and preservice teachers to use the Concept Mastery Routine. One, 
called the "Virtual Workshop," was a computer-based, hypermedia program created 
using the software programs Authorware (Macromedia, Inc.) and Premier 
(Adobe, Inc.). The Virtual Workshop was stored on a recordable compact disc. 
Organized like a book, the content for the Virtual Workshop was comprised of nearly 
100 "electronic" pages. These pages were grouped into 6 chapters, and the chapters 
were grouped into 4 sections. The titles for these sections and chapters were listed in a 
table of contents that was always present along the right-hand quarter of the computer 
screen (see Figure 2 for the names of these sections and their related chapters.). Using 
a mouse, a teacher could select any section or chapter, at any time and as often as 
wanted, by "clicking" its title. Upon doing so, information about the selected section 
or chapter appeared in the presentation window — the remaining three-quarters of the 
screen. This information was in the form of text, video, audio, animated graphics, or a 
combination of these media. For example, immediately after clicking on Chapter 2 
from the table of contents, the chapter's title, "The Linking Steps," appeared in the 
center of the presentation window, and a narrator's voice provided an advance 
organizer about the chapter's contents. Following this advance organizer, the chapter's 
first page appeared in the presentation window. This page contained a brief passage of 
text describing the first Linking Step. Imbedded within this passage were several blue 
bold-faced words. Unlike other words, these words, called hypertext, could be 
"clicked" by moving the cursor to the word and pressing down once on the mouse. 
When clicked, additional information about that word appeared in the presentation 
window. 
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In addition to text, the page also contained a video segment showing a teacher 
performing the first Linking Step with students, and it could be played over and over 
again. To electronically control the video segments, a "remote control" was present on 
the computer screen. Areas on this remote control could be clicked to either play, 
rewind, fast forward, or pause the video segment. 

To enable the learner to electronically "turn" the chapter’s pages, four arrows 
were present in the lower-right-hand comer of the computer screen. The right- 
pointing arrow could be clicked to turn to the chapter's next page, while the left- 
pointing arrow could be clicked to turn to a previous page. The remaining arrows 
could be clicked to turn immediately to either the chapter's first page or last page. 

Like the first page, all but the last page in Chapter 2 contained text describing 
a particular Linking Step, hypertext, and a video segment. The last page, designed to 
check teachers' understanding of Chapter 2, contained three multiple-choice 
questions. Each question could be answered by clicking one of three options. If a 
question was answered correctly, a star appeared next to the question. If a question 
was answered incorrectly, corrective feedback was provided, and the person was 
prompted to answer the question again. Once all three questions on the page were 
answered correctly, a checkmark appeared in the table of contents, next to the title for 
Chapter 2. This checkmark indicated that the chapter had been completed. 

The format of each remaining section and chapter of the Virtual Workshop 
followed a structure similar to that in Chapter 2. However, in these sections and 
chapters, teachers learned about other components of the Concept Mastery Routine, 
practiced completing sample Concept Diagrams, studied the validation research, and 
examined numerous Concept Diagrams constructed by other teachers. Also, teachers 
practiced making decisions about the use of the Concept Mastery Routine in a 
scenario-based simulation. That is, participants read a scenario describing a lesson in 
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which the routine was to be used with students. At designated points in the scenario, a 
multiple -choice question asking participants to decide how to apply the routine was 
presented. Once the question was answered correctly, participants were allowed to 
continue through the scenario. If participants answered the question incorrectly, 
corrective feedback was provided and they were allowed to attempt answering the 
question again. 

Overall, the Virtual Workshop integrated many of the known principles of 
effective teacher development. For example, in the program, teachers were provided: 
(a) rationales which explained why the routine should be used; (b) a thorough 
description of the routine, (c) several models demonstrating how to use the routine in 
a classroom lesson; (d) the opportunity to practice constructing Concept Diagrams 
and to receive corrective feedback; and (e) the opportunity to practice answering 
question about how to apply the routine in a simulated lesson and to receive feedback. 
Also, as will be described latter, following the Virtual Workshop, participating 
inservice teachers made a written commitment to apply the routine. 

The development of the CD-ROM containing the Virtual Workshop cost an 
estimated $40,800. Costs were accrued in four primary areas: hardware, software, 
personnel, and video production. The hardware consisted of: (a) two PowerMac^M 
8500/120av ($3,500 each); (b) two 20 inch Macintosh^^ monitors with speakers 
($905 each); and (c) one JVC^^ CD-ROM recorder ($4,000). The software consisted 
of: (a) one copy of Macromedia^^ AuthorwareTM ($4,995 each); and (b) one copy 
of Adobe^^ Primer TM ($495 each). The personnel and programming time needed 
for development were: (a) one programmer ($30 per hour) for 450 hours; and (b) one 
graphics designer ($30 per hour) for 150 hours. The video production costs were 
comprised of a video-production crew using one Sony^^ Betacam^M recorder for 
three days ($1,500 per day). 





Actual Workshop. The second teacher development program was called the 
"Actual Workshop." Unlike the Virtual Workshop, the Actual Workshop followed a 
more traditional format; it was done live and was presenter-directed, not user- 
directed. That is, the content of the Actual Workshop was presented by a live expert 
using a lecture format. For example, when teaching about the Linking Steps, the 
presenter first stated an advance organizer. Then he displayed and summarized an 
overhead transparency describing the first Linking Step. Next, a video segment of a 
teacher performing the first Linking Step was shown on a monitor. Then, in the same 
manner, the presenter described the remaining six Linking Steps. Finally, three 
multiple-choice questions were presented, and teachers answered them independently, 
were given the correct answers, and were asked if clarification was needed. Once the 
Linking Steps had been covered, the presenter proceeded to other content. Since a 
presenter was directing the Actual Workshop, if a teacher wanted to spend additional 
time reviewing the Linking Steps, he/she could not. Additionally, after learning about 
the Linking Steps, a teacher could not proceed to content of his/her choice. No part of 
the presentation was presented; however, any questions participants asked were 
answered. 

The content covered in the Virtual Workshop was also covered in the Actual 
Workshop. Moreover, the content of the Actual Workshop was organized according 
to the Virtual Workshop's table of contents. To help ensure consistency across 
workshops, each page of the Virtual Workshop, including chapter questions, sample 
Concept Diagrams, and validation research data were downloaded and printed as 
overhead transparencies for the Actual Workshop. Moreover, all the video segments 
shown in the Virtual Workshop were integrated into the Actual Workshop. Finally, 
like students in the Virtual Workshop, students in the Actual Workshop practiced 
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completing sample Concept Diagrams and practiced making decisions about how to 
use the routine in a scenario-based simulation. 

To check content consistency across the two workshops, a content checklist 
(see Appendix A) was created and was completed by an independent rater for both the 
Virtual and Actual Workshops. The rater completed the Virtual Workshop on the 
computer and attended an Actual Workshop. Findings indicate that 97.6% of the 
content presented in the two workshops was the same. 

The development cost of the Actual Workshop was an estimated $9,290. 

These costs were accrued in the same four areas as the Virtual Workshop: computer 
hardware, computer software, personnel, and video production. The computer 
hardware needed to produce the graphic layouts for the overhead transparencies was: 
(a) one PowerMac 61(X)/60 ($1,5(X) each); and (b) one 15 inch Macintosh^M 
color monitor ($395). The computer software required was Adobe^^ Pagemaker’^ 
($395). One teacher trainer was required ($30 per hour) for 100 hours to plan the 
presentation and prepare overhead transparencies. The video production costs were 
comprised of a video-production crew using one Sony"^ Betacam"^ recorder for 
three days ($ 1 ,500 per day). 

Measurement Instruments 

Knowledge Test. A set of seven short-answer questions was developed to test 
participants' understanding of the Concept Mastery Routine (see Appendix B). These 
questions related to teachers' understanding of the Concept Mastery Routine's 
components and procedures. The instructions indicated that participants had 30 
minutes to answer the questions and that only the content of each answer would be 
scored, not spelling, grammar, or punctuation. 

To score participants' answers, evaluation guidelines specifying acceptable 
answers for each question were developed (see Appendix B). For five of the seven 
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questions, participants received up to 3 points for each written answer. For Question 
#2, participants received up to 3 points for the answer to each of the question's three 
parts. For Question #3, participants received up to 2 points for the answer to each of 
the question's seven pans. Unacceptable answers were awarded zero points. Overall, 
participants could earn a maximum score of 38 points. Once a point value had been 
assigned to each answer, the points were totaled, divided by 38, and multiplied by one 
hundred, producing a percentage score. This percentage score was called the 
Knowledge Score. 

Diagram Test. For this test, participants filled in a Concept Diagram for a 
familiar concept, automobile (see Appendix B). This measure was developed to test 
participants' knowledge of what type of information (e.g., characteristic, example, 
etc.) belonged in each of the Concept Diagram's sections. Participants were allowed 
10 minutes to complete the twenty-six blanks on the diagram and were told that only 
the content of their written responses would be scored, not spelling, grammar, or 
punctuation. 

To score participants' completed Concept Diagrams, evaluation guidelines 
specifying acceptable responses were developed (see Appendix B). For twenty-one of 
the Concept Diagram's twenty-six blanks (e.g.. Always Characteristics, Nonexamples, 
Definition, etc.), participants received 5 points for each acceptable response. For the 
remaining blanks (e.g.. Key Words), participants received 1 point for each acceptable 
response. All unacceptable responses were awarded zero points. Overall, each 
participant could earn a maximum of 1 10 points. Once a point value had been 
assigned to each section, the points were totaled, divided by 1 10, and multiplied by 
one hundred, producing a percentage score. This percentage score was called the 
Diagram Score. 
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Implementation Checklist. An observational checklist was developed to assess 



some participants' implementation of the Concept Mastery Routine during a 
classroom lesson (see Appendix B). The checklist was comprised of three sections 
which corresponded with the routine's Cue-Do-Review Sequence. Using section one, 
observers checked teachers' use of behaviors associated with introducing or "Cueing" 
the routine (e.g., gaining students' attention). Using section two, observers checked 
teachers' use of behaviors associated with "Doing" a Concept Diagram with students 
(e.g., naming the targeted concept). Using section three, observers checked teachers' 
use of behaviors associated with "Reviewing" the concept (e.g., asking procedural 
questions). A total of thirty-nine teacher behaviors were assessed. For twenty-four of 
these behaviors, teachers received five points per behavior when it was performed. 
Regarding the remaining fifteen behaviors, teachers received one point when each 
was performed. If any behavior was not performed, the teacher received zero points 
on the checklist for that behavior. Additionally, each time the teacher interacted with 
a student in conjunction with one of the thirty-eight behaviors, an additional point, up 
to twenty points in total, was awarded. 

Overall, each time the Concept Mastery Routine was used, teachers could earn 
155 points on the checklist. Once a point value had been assigned to each behavior on 
the checklist by an observer, the points were totaled, divided by 155, and multiplied 
by one hundred, producing a percentage score. This percentage was called the 
Implementation Score. 

To guide observers' use of the checklist, observational guidelines were 
developed (see Appendix B). These guidelines objectively defined each behavior and 
specified how points on the checklist were to be awarded. The observers practiced 
scoring videotaped presentations of the routine until the observers agreed on at least 
90% of their recordings on the checklist. 
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Implementation Recording Form. A recording form was developed to 
determine a teacher's willingness to implement the routine (see Appendix B). On this 
recording form, following training in the Concept Mastery Routine, participants were 
asked to identify a date by which they anticipated they would use the routine for the 
first time. Several weeks later, these teachers were contacted and asked if they had 
implemented the routine, and if so on what date. Presumably, if they were 
comfortable with the routine, teachers would be more likely to implement it soon 
after the workshop. A teacher's scores on this measure were the number of school 
days from training to predicted implementation and the number of school days from 
training to actual implementation. These scores were called the Latency of 
Implementation Scores. At the end of the study, the teachers were also asked how 
many times they had implemented the routine in their classes. 

Satisfaction Questionnaire. A fourteen-item questionnaire was developed to 
assess participants' satisfaction with the training they received (see Appendix B). 
Each questionnaire item included a 7-point Likert-type scale ranging from "disagree" 
(1) to "agree" (7). The items were designed to determine: (a) how enjoyable 
participants found the training; (b) how engaged participants felt during the training; 
(c) how understandable participants found the content, and (d) how applicable 
participants found the content. Teachers' ratings for these items were called the 
Satisfaction Scores. 

Reliability. Interscorer reliability was determined by having two scorers 
independently score 20% of the Knowledge Tests and Diagram Tests. Interscorer 
reliability for the Implementation Checklist was determined by having two observers 
simultaneously record information in 20% of the classroom observations of teachers' 
implementation of the routine. The points awarded by the two observers were 
compared item-by-item for each of the measures. The percentage of agreement was 
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calculated by dividing the number of agreements by the number of disagreements and 
multiplying by 100. For the Knowledge Tests, the scorers agreed 1397 times out of 
1444 opportunities to agree (total percentage of agreement = 96.75%). For the 
Diagram Tests, the scorers agreed 967 times out of 988 opportunities to agree (total 
percentage of agreement = 97.87%). For the Implementation Checklists, the scorers 
agreed 1979 times out of 2040 opportunities to agree (total percentage of agreement = 
97.01%). For the Implementation Recording Form, the scorers agreed 16 times out of 
16 opportunities to agree (total percentage of agreement = 100%). 

Virtual Workshop Procedures 

Preservice Teachers. Twenty-nine of the 58 preservice teachers were 
randomly assigned to the Virtual Workshop group. In groups of ten or fewer, these 
preservice teachers participated in a Virtual Workshop on the Concept Mastery 
Routine. Immediately before the Virtual Workshop, a session leader pretested the 
preservice teachers on the Knowledge and Diagram Tests. Participants were allowed a 
maximum of 30 minutes to complete the Knowledge Test and a maximum of 10 
minutes to complete the Diagram Test. If participants asked about a question’s 
answer, they were instructed to answer the question as best they could. Next, using a 
computer attached to a large-screen monitor, the session leader loaded the Virtual 
Workshop and provided a five minute demonstration of how to "page" or navigate 
through the computer program. Following this demonstration, the preservice teachers 
were escorted to an adjacent computer lab. Each preservice teacher then selected a 
computer on which the Virtual Workshop was loaded and began to navigate the 
computer program. The preservice teachers were provided a maximum of two and a 
half hours to navigate the entire program. The session leader remained in the lab to 
provide technical support (e.g., restart frozen computers, adjust computer volume, 
change broken CD-ROM drives). If preservice teachers had questions about content. 
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they were told the computer program contained all the information they needed to 
understand the Concept Mastery Routine. Once the preservice teachers had completed 
the Virtual Workshop, they were administered the Knowledge Test, Diagram Test, 
and Satisfaction Questionnaire. Again, preservice teachers were provided a maximum 
of 30 minutes for the Knowledge Test, and 10 minutes for the Diagram Test. No time 
limit was set for the Satisfaction Questionnaire. Altogether, a total of four hours were 
scheduled for the preservice teachers to complete the pretests, the Virtual Workshop, 
and the posttests. 

Inservice Teachers. Of the 10 inservice teachers, five were randomly assigned 
to the Virtual Workshop treatment group. These four general education teachers and 
one special education teacher participated in the Virtual- Workshop on the Concept 
Mastery Routine following the same procedures as those used with preservice 
teachers. However, before participating, the four general education teachers were 
observed delivering three or more lessons in their classrooms. During these lessons, 
each participating teacher delivered instruction on a concept of his/her choice. The 
special education teacher was not observed. During each lesson, observers scored the 
general education teachers' presentation using the Implementation Checklist. Once the 
baseline data were stable or showed decreasing trends for each of these teachers, they 
participated the Virtual Workshop on the Concept Mastery Routine. Afterward, these 
four general education teachers were again observed presenting lessons in which they 
indicated a concept would be taught. For example, during each lesson, observers 
again scored the teachers' presentation using the Implementation Checklist. She 
teacher did not implement the routine. The special educator taught at a school with 
several of the participating general educators, and she wanted to leant about the 
routine these general educators were going to use in classes in which her 
mainstreamed special education students were enrolled. 
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Actual Workshop Procedures 



Preservice Teachers. Twenty-nine preservice teachers were randomly assigned 
to the Actual Workshop treatment group. Like the preservice teachers assigned to the 
Virtual Workshop treatment group, these preservice teachers attended the Actual 
Workshop in groups of ten or fewer and were initially pretested on the Knowledge 
and Diagram Tests. Once the pretests were completed, a session leader began the 
Actual Workshop. Using overhead transparencies, an overhead projector, videotape 
segments, and a videotape player attached to a large-screen monitor, the session 
leader directed the Actual Workshop. Any questions asked about the Concept 
Mastery Routine were answered. Once all the content had been covered, all activities 
had been completed, and all questions had been answered, each preservice teacher 
was administered the Knowledge Test, Diagram Test, and Satisfaction Questionnaire. 
Altogether, following the same timeline as the Virtual Workshop, a total of four hours 
were scheduled for the completion of the pretests, the Actual Workshop, and the 
posttests. 

Inservice Teachers. Five inservice teachers were randomly assigned to the 
Actual Workshop treatment group. These four general education teachers and one 
special education teacher participated in the Actual Workshop on the Concept 
Mastery Routine following the same procedures as those used with preservice 
teachers. Moreover, like the general education teachers in the Virtual Workshop 
treatment group, before Md after training in the Concept Mastery Routine, these 
inservice teachers were observed in their classrooms presenting lessons in which they 
indicated a concept would be taught. Observers scored these lessons using the 
Implementation Checklist. Again, like the special education teacher attending the 
Virtual Workshop, this special education teacher taught at a school with several of the 
participating general educators. She wanted to learn about the routine these general 
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educators were going to use in classes in which her mainstreamed special education 
students were enrolled. She did not implement the routine. 

Experimental Designs 

Four experimental designs were employed simultaneously during this study. A 
pretest-posttest control-group design (Campbell & Stanley, 1963) was used to 
compare the Knowledge Scores and Diagram Scores of preservice teachers 
participating in the Virtual and Actual Workshops and of inservice teachers in both 
workshops. A posttest-only control-group design (Campbell & Stanley, 1963) was 
used to compare the Satisfaction Scores of preservice teachers participating in the 
Virtual and Actual Workshops, and of inservice teachers in both workshops. To 
compare inservice teachers' Latency of Implementation Scores, a posttest-only 
control-group design was used (Campbell & Stanley, 1963). Finally, to determine the 
effects of the workshops on inservice teachers' Implementation Scores, a multiple- 
baseline across-teachers design (Baer, Wolf, & Risley, 1968) was employed and 
replicated two times. 
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Chapter 4 
Results 

Preservice Teachers 

Listed in Table 1 are the mean percentage scores and standard deviations 
summarizing the pretest and posttest performances of preservice teachers from both 
treatment groups on the Knowledge and Diagram Tests. To compare the differences 
between these pretest and posttest scores within each treatment group, t-tests were 
performed and indicated that: (a) the posttest scores of preservice teachers who 
participated in the Virtual Workshop were significantly higher than their pretest 
scores on the Knowledge Test [i (28) = 15.35, p < 0.00] and the Diagram Test [l (28) 
= 37.99, p < 0.00]; and (b) the posttest scores of preservice teachers who participated 
in the Actual Workshop were significantly higher than their pretest scores on the 
Knowledge Test [t (28) = 17.05, p < 0.00] and the Diagram Test [i (28) = 38.70, p < 
0 . 00 ]. 

To determine whether the two training methods had differential effects on the 
preservice teachers' performance, analyses of covariance (ANCOVA) were employed 
using the preservice teachers’ posttest scores as the dependent variable and their 
pretest scores as the covariate. These analyses revealed no statistically significant 
difference between the posttest scores that preservice teachers in the Virtual and 
Actual Workshops earned on the Knowledge Test [F (1, 55) = 0.44, p < 0.51] or the 
Diagram Test [F (1, 55) = 0.00, p < 0.98]. 

Figure 3 depicts the results from the Satisfaction Questionnaire distributed to 
all preservice teachers who participated. Overall, preservice teachers rated both the 
Virtual and Actual Workshops favorably. Across all fourteen Likert-scale items, mean 
Satisfaction Scores ranged from 5.10 to 6.14 for preservice teachers participating in 
the Virtual Workshop and from 4.93 to 6.66 for preservice teachers participating in 
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the Actual Workshop. To illuminate differences between the mean Satisfaction Scores 
of each treatment group for each item, analyses of variance (ANOVA) were 
performed. These analyses indicated that teachers who participated in the Actual 
Workshop rated questionnaire items #1 [F (1, 56) = 8.38, p < 0.01], #5 [F (1, 56) = 
5.09, p < 0.03], and #9 [F (1, 56) = 5.72, p < 0.02] significantly higher than 
preservice teachers who participated in the Virtual Workshop. Interestingly, each of 
these three items pertained to how well the preservice teachers thought they 
understood the content presented. The remaining questionnaire items, which related 
to how applicable participants found the content, how enjoyable they found the 
training, and how engaging they found the training, were rated similarly by preservice 
teachers from both groups. 

Inservice Teachers 

Listed in Table 2 are the mean percentage scores and standard deviations 
summarizing the pretest and posttest performances of inservice teachers from both 
treatment groups on the Knowledge and Diagram Tests. The differences between 
these pretest and posttest scores within each treatment group were compared using 
Wilcoxon Signed Ranks Tests. These analyses indicated: (a) the posttest scores of 
inservice teachers who participated in the Virtual Workshop were significantly higher 
than their pretest scores on the Knowledge Test [z = 2.19, p < 0.04] and on the 
Diagram Test [z = 2.19, p < 0.04]; and (b) the posttest scores of inservice teachers 
who participated in the Actual Workshop were significantly higher than their pretest 
scores on the Knowledge Test [z = 2.21, p < 0.04] and on the Diagram Test [z = 1.80, 
p<0.04]. 

Though both treatments produced significant improvement, to identify 
possible differential effects of the two training methods on inservice teachers' 
Knowledge and Diagram Scores, Kruskal-Wallis One-Way Analyses of Variance by 
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Ranks (KWANOVA) were used. These analyses revealed no significant differences 
between the pretest scores of teachers participating in the Virtual and Actual 
Workshops on the Knowledge Test [X^ (1, N = 10) = 0.05, p < 0.83] and the Diagram 
Test [x 2(1,N = 10) = 2.53, p< 0.11], or between the posttest scores of these same 
teachers on the Knowledge Test [X^ (1, N = 10) = 0.18, p < 0.67] and the Diagram 
Test [X^ (1, N = 10) = 2.22, p < 0.14]. Interestingly, the pretest Diagram Scores of 
inservice teachers who participated in the Actual Workshop were 32.95 percentage 
points higher than those of the inservice teachers who participated in the Virtual 
Workshop. Given these different levels of prior knowledge, one would assume the 
Actual Workshop teachers would also score higher than Virtual Workshop teachers at 
posttest. However, on average, the inservice teachers wJio participated in the Virtual 
Workshop actually scored higher than teachers who participated in the Actual 
Workshop on the Diagram Posttest. 

Eight of the inservice teachers were observed in their classrooms teaching 
concepts to their students. Figures 4, 5, 6, and 7 show the performance of these 
teachers as recorded on the Implementation Checklist. In these figures, each teacher's 
performances before participating in a workshop (i.e., during baseline) are shown to 
the left of the solid vertical line and their performances after participating in 
workshop (i.e., after training) are shown to the right of this line. The performances of 
teachers who participated in the Virtual Workshop are shown in Figures 4 and 5, 
whereas the performances of teachers who participated in the Actual Workshop are 
shown in Figures 6 and 7. 

As illustrated in these two figures, during baseline, the percentage of points 
earned on the Implementation Checklist by teachers who participated in the Virtual 
Workshop ranged from 0% to 31.60% (M = 12.45%, SD = 9.15); the percentage of 
points earned by teachers who participated in the Actual Workshop ranged from 
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3 . 20 % to 38.06% (M = 19.03%, SD = 1 1.82). The scores earned during baseline by 
teachers in both treatments were compared using the KWANOVA . Results revealed 
no significant differences between the groups' scores [X^ (1, N = 8) = 3.43, p = 0.06]. 

After training, the percentage of points earned on the Implementation 
Checklist by teachers who participated in the Virtual Workshop ranged from 73% to 
92.25% (M = 84.68%, SD = 5.54) (See Figures 5 and 6). Moreover, 10 of the 12 
lessons presented by teachers after training exceeded the arbitrary mastery level of 
80%. Visual examination of scored Implementation Checklists suggested that the 
teachers rarely reviewed what had been learned with their students. These teachers 
also did not regularly help students discuss the characteristics possessed by examples 
of the targeted concept. Moreover, after the first time teachers used the routine, they 
rarely explained to students how the routine would help them learn and how they 
were to participate when it was used. 

The percentage of points earned on the Implementation Checklist after 
training by teachers who participated in the Actual Workshop ranged from 58.70% to 
100% (M = 78.25%, SD = 13.27) (See Figures 6 and 7). Of the 12 lessons presented 
by these teachers after training, five exceeded the mastery level of 80%. Visual 
examination of the Implementation Checklists suggested that, like teachers who 
participated in the Virtual Workshop, these teachers also rarely reviewed with their 
students what had been learned. Moreover, these teachers also did not regularly help 
students to discuss the characteristics possessed by examples of the targeted concept. 
One teacher who participated in the Actual Workshop lost some points on the 
Implementation Checklist because he did not always fill in the example and 
nonexample sections of the Concept Diagram; rather, he stated the examples and 
nonexamples aloud. On other occasions, teachers did not earn points because they did 
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not provide students an adequate number of characteristics or examples of the 
targeted concept. 

The KWANOVA was used to compare the scores earned after training by 
teachers in the two treatment groups. No statistically significant differences between 
the groups' scores were revealed [X^ n = 8) = 1.62, p = 0. 23]. 

Immediately after training, the eight inservice teachers predicted a date on 
which they would first implement the routine. Later, they reported the date on which 
they actually implemented the routine. The results are depicted in Figure 8 and 
indicate that teachers who participated in the Actual Workshop predicted they would 
implement the Concept Mastery Routine in fewer days than teachers who participated 
in the Virtual Workshop. Teachers who participated in the Actual Workshop 
predicted they would implement the routine after an average of 8.25 school days (SD 
= 2.63), whereas teachers who participated in the Virtual Workshop predicted they 
would implement the routine after an average of 17 school days (SD = 8.60). Despite 
the variation among these predictions, a KWANOVA revealed that these differences 
were not statistically significant [X^ n = 8) = 2.55, p = 0.1 1]. 

With regard to actual implementation, the teachers who participated in the 
Actual Workshop implemented the routine sooner than teachers who participated in 
the Virtual Workshop. On average, they implemented the routine after 19.50 days 
(SD = 16.09), whereas teachers who participated in the Virtual Workshop 
implemented the routine after an average of 29 school days (SD = 7.79). Again, 
however, the KWANOVA revealed that these differences were not statistically 
significant [X^ (1, N = 8) = 0.75, p = 0.39]. Interestingly, for both groups, the number 
of days between the dates teachers predicted they would implement and actually 
implemented the routine were very similar. That is, teachers who participated in the 
Actual Workshop, on average, implemented the routine 11.25 school days (SD = 
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15.71) after the date they had predicted, whereas teachers who participated in the 
Virtual Workshop, on average, implemented the routine 12 school days (SD = 1 1.75) 
after the date they had predicted. 

Depicted in Figure 9 are the results from the Satisfaction Questionnaire for 
participating inservice teachers. Overall, like the preservice teachers, the inservice 
teachers rated both the Virtual and Actual Workshops favorably. In fact, the mean 
Satisfaction Scores of the inservice teachers were even higher than those of the 
preservice teachers. Across all fourteen Likert-scale items, mean Satisfaction Scores 
ranged from 5.50 to 7.00 for inservice teachers who participated in the Virtual 
Workshop and the Actual Workshop. Analysis of the teachers' mean Satisfaction 
Scores using the KWANOVA revealed no significant differences between the two 
groups on any item. Thus, what was most notable about these findings was how 
similarly teachers participating in the two workshops rated each item. 
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Chapter 5 
Discussion 

Conclusions and Relationship to Previous Research 

The purpose of this project was to develop an interactive multimedia program 
and to examine: a) its effects on preservice and inservice teachers' knowledge of an 
instructional innovation; b) its effects on inservice teachers' skill in implementing the 
innovation in their classrooms with students; and c) to outline the costs associated 
with its development. 

Several conclusions can be drawn from the results of this research effort. First, 
compared to their pretest scores, the posttest scores preservice and inservice teachers 
earned on the Knowledge and Diagram Tests significantly improved following 
participation in either the Virtual or Actual Workshops. Unfortunately, the magnitude 
of their improvement on the Knowledge Test was not as large as might be desired. 
That is, despite improving an average of at least 45 percentage points from pretest to 
posttest, preservice and inservice teachers' posttest scores on the Knowledge Test still 
averaged from only 49.09% to 63.68%. The instruction, models, practice, and 
feedback these teachers received did impact their knowledge about the innovation, but 
the teachers' recall of information related to the Concept Mastery Routine was 
approximately 40% incomplete or inaccurate after training. Nevertheless, this 
magnitude of gain on the Knowledge Test was considerably higher than the 
magnitude of gain reported by other investigators utilizing interactive multimedia 
programs (Lanza & Roselli, 1991; Quadc, 1993; Santer et al., 1995; Schank & Rowe, 
1993). 

The magnitude of the effects of both workshops on preservice and inservice 
teachers' performance on the Diagram Test, on the other hand, was socially 
significant. Overall, the teachers' average scores ranged from 84.14% to 93.63%. For 
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this test, participating teachers were provided a blank Concept Diagram and asked to 
complete it for the targeted concept "automobile." The graphic stmcture of the blank 
Concept Diagram was designed to prompt the entry of specific content related to the 
targeted concept. Thus, because of the prompts the Concept Diagram provided, this 
test may have not been as difficult as a Knowledge Test which required complete 
recall. Nevertheless, given that teachers improved, on average, at least 45 percentage 
points, this finding indicates that the instmction, practice, and feedback these teacher 
received led them to construct highly accurately Concept Diagrams. 

Second, both the Virtual Workshop and the Actual Workshop had similar 
effects on the scores earned by preservice and inservice teachers on the Knowledge 
and Diagram Tests. Thus, for these two measures, the Virtual and Actual Workshops 
appear to be interchangeable. Interestingly, other studies comparing the effects of 
user-directed interactive multimedia programs to computer-directed tutorials 
(D'Alessando et al., 1992; Hudson & Holland, 1992) and teacher-directed lectures 
(Lanza & Roselli, 1991; Quade, 1993; Santer et al., 1995; Schank & Rowe, 1992) 
have also contained similar findings. 

Third, an analysis of the scores from the actual implementation of this 
innovation suggests that inservice teachers who participated in the Virtual Workshop 
and those who participated in the Actual Workshop performed a substantially greater 
number of the Concept Mastery Routine's targeted behaviors after training than before 
training. Moreover, these findings also suggest that, on average, inservice teachers 
who participated in the Virtual Workshop performed the Concept Mastery Routine in 
a manner similar to those teachers who participated in the Actual Workshop. This 
finding is important, for other studies have not measured the effect of an interactive 
multimedia program on adults' ability to apply a sophisticated skill in an authentic 
setting like a classroom. Still, despite the marked improvement of both groups with 
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regard to teaching concepts, room remained for all participating teachers to use the 
routine more completely. For example, inservice teachers who participated in the 
Virtual Workshop rarely reviewed what had been learned with their students, did not 
regularly discuss what characteristics, examples, and nonexamples of the targeted 
concept possessed, and often failed to explain to students how the routine would help 
them to learn. 

Interestingly, a previous study that measured teachers' ability to implement the 
Concept Mastery Routine following a workshop format similar to the Actual 
Workshop revealed that, on their first try, participating teachers could implement the 
routine nearly perfectly (Bulgren, Schumaker, & Deshler, 1988). This workshop 
differed from the Actual and Virtual Workshops in one important way: before trying 
the routine with students, participating teachers first taught a practice lesson to other 
teachers and received corrective feedback on their performance. During the Virtual 
and Actual Workshops in this study, teachers answered questions in a scenario-based 
simulation only; they did not have an opportunity to actually practice the routine in a 
live simulation activity. 

Fourth, although the inservice teachers in both groups performed similarly on 
the Knowledge Test, the Diagram Test, and the Implementation Checklist, some 
variation existed between their predicted and actual dates of implementation on the 
Implementation Recording Form. Though these differences were not statistically 
significant, they do suggest that inservice teachers who participated in the Actual 
Workshop were more comfortable with their understanding of the Concept Mastery 
Routine than teachers who participated in the Virtual Workshop. 

Fifth, preservice and inservice teachers' satisfaction ratings of the Virtual and 
Actual Workshops were favorable. The fact that preservice and inservice teachers 
rated the Virtual Workshop favorably was not surprising. Interactive multimedia 
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programs are typically well-liked by their users. What was surprising, however, was 
how favorably preservice and inservice teachers rated the Actual Workshop. In three 
previous studies in which the consumer satisfaction ratings of adults who participated 
in group lectures and adults who participated in interactive multimedia programs were 
compared, the adults who experienced the multimedia programs rated their enjoyment 
of the interactive multimedia programs substantially higher than those who 
participated in lecture (Hudson & Holland, 1992; D’Alessandro et al., 1992; Santer et 
al., 1995). Interestingly, preservice teachers who participated in the Actual Workshop 
had significantly higher ratings on questionnaire items pertaining to how well they 
understood the content presented than preservice teachers who participated in the 
Virtual Workshop. This finding corroborates the finding that Actual Workshop 
participants predicted that they would use the Routine sooner than Virtual Workshop 
participants. Such findings may be cause for concern because if Virtual Workshop 
participants walk away from a session feeling uncomfortable about using an 
innovation, they may never use it. Since both groups of inservice teachers understood 
that the researchers would be visiting their classrooms to observe their 
implementation of the Routine, whether or not they would have actually implemented 
it in their classrooms if visitors had not been coming to observe is unknown. 

Finally, the development cost of the Virtual Workshop was nearly four times 
as expensive as the development cost of the Actual Workshop. Still, though the 
development cost of the Virtual Workshop was greater, the cost to implement this 
workshop with teachers is less than the cost to implementation the Actual Workshop. 
That is, each time the Actual Workshop is implemented, $250 for the half-day 
presenter and $30 for each participating teacher's half-day substitute needs to be 
spent. Thus, the cost for 100 teachers to attend an Actual Workshop would be $3,250. 
However, to implement the Virtual Workshop with one teacher could cost as little as 
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$2.50, the price of one compact disc with postage. Thus, training for 100 teachers 
could cost only $250.00. Conceivably, if the number of teachers trained was 
increased, the cost to develop and implement the Virtual Workshop could be similar 
to or even less than the cost to develop and implement the Actual Workshop. 

This research has both replicated and extended the literature on interactive 
multimedia. First, it has replicated existing research by: (a) demonstrating the effects 
of an interactive multimedia program on preservice and inservice teachers' knowledge 
of an instructional innovation; and (b) comparing their satisfaction with the 
interactive multimedia program and a traditional professional development program. 
Second, this study has extended existing research by measuring the effects of an 
interactive multimedia program on inservice teachers' skill in applying an 
instructional innovation in authentic classroom situations. In previous studies 
participants only applied what they had learned to paper-pencil tasks (Patterson & 
Yaffe, 1993; McGrath, 1992) or simple procedural tasks like folding paper cranes 
(Shyu & Brown, 1991). This study has provided evidence that inservice teachers can, 
on average, apply a complex instructional innovation in their own classrooms with 
84.68% accuracy after completing an interactive multimedia program. Finally, this 
study has shed light on the cost associated with interactive multimedia programs. 
Though a comprehensive cost-benefit analysis is not feasible at this time, the 
information regarding costs provides some guidance about the potential use of 
interactive multimedia programs for teacher development in the future. 

Limitations 

Despite these encouraging findings, this study is limited in several ways. First, 
only the inservice teachers were observed implementing the Concept Mastery Routine 
with students. Whether preservice teachers could correctly implement the routine in 
an authentic setting at levels similar to the inservice teachers remains unknown. 
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Second, in most cases, inservice teachers who participated in either of the two 
workshops were, in most cases, observed using the routine only three times. Although 
the inservice teacher's scores on the Implementation Checklist were consistent, 
whether this level of consistency would be maintained over a semester or school year 
is unknown. Moreover, whether participating teachers would continue to use the 
Concept Mastery Routine is unclear. For example, one teacher indicated that he had 
used the routine with 12 different concepts during a three-month period, whereas 
another teacher implemented the routine only two times in this same period of time. 
Though this second teacher used the routine adequately, she was uncomfortable using 
it with her students and chose to not use the routine again. 

Third, all of the teachers were given a pretest prior to participating in either 
the Virtual or Actual Workshop. This pretest may have sensitized teachers to 
particular content. As a result of being sensitized, their performance on the posttests 
and in the classroom may have been affected. Neither the Virtual or Actual Workshop 
may have similar effects with another group of preservice or inservice teachers if 
those teachers are not first given a pretest. 

Fourth, the only instructional innovation taught to the participating preservice 
and inservice teachers through the Virtual Workshop was the Concept Mastery 
Routine. Thus, the effects of the Virtual Workshop on these teachers' performance 
may have resulted from the novelty of this approach. Whether these same teachers 
would benefit as much from participating in a second or third Virtual Workshop on 
other instructional innovations is unknown. Whether teachers can learn about other 
instructional innovations through this method is also unknown. 

Finally, all of the participants in this study were volunteers. These teachers 
may not be representative of the general population of preservice and inservice 
teachers. Neither the Virtual or Actual Workshops may have similar effects with 
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another group of preservice or inservice teachers if those teachers are required to 
participate. 

Future Research 

Additional research is needed before the Virtual Workshop for the Concept 
Mastery Routine can be used confidently in teacher development programs. Clearly, 
given the low posttest Knowledge Scores of inseivice teachers who completed the 
Virtual Workshop, studies need to be conducted to examine how to improve the 
understanding and recall of critical content presented in interactive multimedia 
programs. Secondly, studies should also be conducted to identify methods for 
improving the ability of teachers who have completed the Virtual Workshop to 
implement the Concept Mastery Routine more precisely. Although the teachers in the 
study did apply the Concept Mastery Routine with over 80% accuracy, in an earlier 
study (Bulgren et al., 1988) teachers who completed a traditional workshop 
implemented the routine nearly perfectly with students on their first try. This 
traditional workshop differed from the Virtual Workshop in that participating teachers 
taught a practice lesson to one another and received corrective feedback on their 
instruction. Perhaps by following up the Virtual Workshop with a similar 
combination of practice and feedback, teachers participating in the Virtual Workshop 
could apply the routine nearly perfectly with students on their first try, too. 

Once improvements to the Virtual Workshop are made, long-term studies in 
which teachers learn a series of instructional innovations using such an interactive 
multimedia program should be conducted. Such studies would allow investigators to 
determine whether the novelty of interactive multimedia impacts teachers' 
professional development. Moreover, such a study would enable investigators to 
determine whether teachers can sustain the application of newly learned innovations 
over time. In conjunction with this research, a cost-benefit analysis could also be 
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conducted. This analysis would more accurately reflect the true costs associated with 
developing and using interactive multimedia programs like the Virtual Workshop. 
That is, after the first interactive multimedia program was developed, subsequent 
programs would be less expensive to produce because the computer programmers 
would be trained and no additional computer equipment would need to be purchased. 
Thus, through the development of several interactive multimedia programs, a more 
accurate price for each program could be calculated. 

Teacher Development Implications 

Since the findings suggest that the Virtual Workshop can effectively advance 
both preservice and inservice teachers' knowledge of and skill in implementing a 
complex instructional routine in their classrooms with students and are satisfied with 
the instruction, this medium may have great impact on the field of teacher education. 
For example, at the preservice level, interactive multimedia programs could be used 
in place of lectures given in methods courses. That is, education students could learn ■ 
about different instructional methods at home or at a computer lab. Moreover, this 
instruction would provide a degree of depth and breadth not possible in the limited 
time of most university courses. In turn, course time could be used to provide 
preservice teachers a setting in which to practice and receive corrective feedback 
about an innovation learned from an interactive multimedia program. At the inservice 
level, teachers could choose to learn about cutting-edge innovations which they are 
interested in learning, rather than attending traditional workshops about methods 
administrators think they should know. These teachers could also learn about cutting- 
edge innovations from the very best the field of education has to offer, and do so 
according to their own schedule. Moreover, districts could potentially provide 
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teachers such professional development opportunities at a cost that is less per teacher 
than existing professional development models. 

Summary 

The results of this study indicate that compared to their pretest scores, the 
posttest scores earned by inservice and preservice teachers on the Knowledge and 
Diagram Tests significantly improved following participation in either the Virtual or 
Actual Workshops. Moreover, preservice and inservice teachers' satisfaction ratings 
of both the Virtual and Actual Workshops were favorable. Inservice teachers who 
participated in the Virtual Workshop and Actual Workshops correctly performed a 
substantially greater number of the innovation's targeted behaviors after training than 
before training. Overall, both the Virtual and Actual Workshops had similar, positive 
effects on both teachers understanding and implementation of the instructional 
innovation. This study suggests interactive multimedia programs like the Virtual 
Workshop may provide a new medium through which effective teacher development 
can be provided. Potentially, this medium could provide general education teachers, 
who teach academically diverse classes of students, the training in inclusive practices 
they testify they need. 
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Content Checklist for the Virtual and Actual Workshops 



A. Introduction 

Challenges Teachers Face: 

□ Large volume of content to teach 

□ Limited amount of instructional time 

□ High expectations for student achievement 

□ Cultural diversity among students 

□ Academic diversity among students 

The Concept Mastery Routine: 

□ Specify a concept students will learn about 

□ Place ±at concept within a larger framework 

□ Explore students' prior knowledge 

□ Specify characteristics of the concept 

□ Analyze examples and nonexamples of the concept 

B. Overview 
Advance Organizer: 

□ The three parts of the Concept Mastery Routine (CMR) will be described 

□ Model lessons of the CMR being used with student will be reviewed 

Part One: The Concept Diagram 
Advance Organizer: 

□ The Concept Diagram (CD) is a graphic device 

□ The CD has seven sections 

CD Section 1 : 

□ Targeted Concept Name (TCN) defined 

□ TCN written in smaller rectangle 

□ Example TCN (rhombus) provided 

CD Section 2: 

□ Overall Concept Name (OCN) defined 

□ OCN written in larger rectangle 

□ Example OCN (quadrilaterals) provided 

CD Section 3: 

□ Key Words defined 

□ Key Words written in large vertical rectangle 

□ Example Key Words for rhombus provided 

CD Section 4: 

□ Define Characteristics 

□ Characteristics written on solid, wavy, and dashed lines 

□ Example Characteristics for rhombus provided 



O 

ERIC 



67 



?2 



CD Section 5: 

□ Examples and Nonexamples defined 

□ Examples and Nonexamples written in ovals 

□ Sample Examples and Nonexamples for rhombus provided 

CD Section 6: 

□ Testing Ground Defined 

□ Testing Ground items written between Examples and Nonexamples 

□ Sample Testing Ground items provided 

CD Section 7: 

□ Concept Definition defined 

□ Concept Definition written in bottom rectangle 

□ Example Definition for Rhombus provided 

CD Summary: 

□ Graphic Device 

□ Seven Sections 

□ Draft created before used 

□ Constructed following linking steps 

CD Questions: 

□ Examples of the Targeted Concept must possess what? 

□ Key Words are words a teacher or student associates with what? 

□ The Concept Definition is a sentence that contains what? 

Part Two: The Linking Steps 
Advance Organizer: 

□ Linking Steps (LS) followed to fill-in the CD 

□ There are seven LS 

□ Together, that first letter firom each LS spells CONCEPT 
LS 1: 

□ Action of Offering the TCN described 

□ CD Section 1 filled 

□ Video model provided 

LS 2: 

□ Action of Offering OCN described 

□ CD Section 2 filled 

□ Video model provided 

LS 3: 

□ Action of Noting Key Words described 

□ CD Section 3 filled 

□ Video model provided 
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LS4: 

□ Action of Classifying Characteristics described 

□ CD Section 4 filled 

□ Video model provided 

LS 5; 

□ Action of Exploring Examples and Nonexamples described 

□ CD Section 5 filled 

□ Video model provided 

LS 6: 

□ Action of Practicing with a New Example described 

□ CD Section 6 filled 

□ Video model provided 

LS7: 

□ Action of Tying Down a Definition described 

□ CD Section 7 filled 

□ Video model provided 

LS Summary; 

□ LS used to involve students in construction of CD 

□ CD must be constructed with student input 

□ Seven LS 

□ Mnemonic CONCEPT 

□ Order of steps may be changed 

LS Questions: 

□ After brainstorming Key Words the teacher and students should do what? 

□ When determining if an item is an Example the student should do what? 

□ The purpose of the LS is what? 

Pan Three: The Cue-Do-Review Sequence 
Advance Organizer; 

□ The Cue-Do-Review Sequence (CDR) are the three phases of instruction a teacher 
follows when using the CMR 

Phase One: 

□ Purpose of the Cue explained 

□ Three parts of Cue described 

□ Video model shown 

Phase Two: 

□ Do defined as the performance of the LS 
Phase Three: 

□ Purpose of the Review explained 

□ Two types of Review questions described 
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□ Video model shown 



Phase Summary: 

□ The CDR are the three phases of instruction a teacher follows when using the 
CMR 

□ Cue p>erformed to gain students attention 

□ Do performed to complete CD with students using LS 

□ Review performed to check understanding 

Phase Questions: 

□ The purpose of the Cue phase of the sequence is what? 

□ To complete the DO phase of the sequence the teacher does what? 

□ During the Review phase of the sequence the teacher asks what? 

C. Instructional Guidelines 

Practice: Getting Ready! 

Advance Organizer: 

□ Opportunity to practice constructing a concept diagram will be provided 
Select Content for the CMR: 

□ Use the CMR to teach concepts important for future learning 

□ Consider the concepts abstractness, organization, relevance, interest, and 
complexity 

Decide When to Use the Routine: 

□ The CD must be constructed in partnership with students 

□ Students need prior knowledge of the concept to participate in the CMR 

□ Consider if students have adequate knowledge 

□ Give students needed prior knowledge before using CMR 

Construct a Draft: 

□ Construct a draft CD before using the CMR 

□ Constructing a draft will help you communicate more clearly 

□ Students provided opportunity to construct a draft and get needed clues and 
answers to complete draft 

Practice: Using the Routine 
Advance Organizer: 

□ Opportunity to use the CMR in a simulated lesson will be provided 
Complete Simulation: 

□ Students provided opportunity to read simulated lesson, answer questions, and 
receive needed clues and answers to complete simulation 

D. Research Findings 

Concept Acquisition: 

□ Students score significantly better on concept acquisition tests 
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□ Students with and without learning disabilities made similar gains 
Unit Tests: 

□ Students scored significantly better on unit tests 

□ Gains for student with and without disabilities were moderate 

□ The percentage of student who passed unit tests also increased 

Note Taking: 

□ Students recorded more items teachers mentioned in notes 

□ Teachers wrote more information on the board 

Successful Teachers: 

□ Attended a four- hour workshop 

□ Discussed using the routine with colleagues 

□ Spent time preparing to use the routine 

□ Taught students how to understand the routine 

□ Used the routine regularly 

E. Library 

History: 

□ Civil War 

□ Democracy 

□ Monopoly 

English: 

□ Story 

□ Nonfiction 

Math: 

□ Fraction 

□ Prism 

□ Square 

Science: 

□ Organism 

□ Herbivore 
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Number: 

Knowledge Test 

The Concept Mastery Routine 

University of Kansas 
Center for Research on Learning 



Name: 



Date: 



Directions: In the 30 minute time limit, responded to each of the statements below. 

Number and record your responses on the paper provided. Please note, 
only the content of your responses will be scored (not spelling, grammar, 
punctuation, etc.) 



1. Briefly describe the purpose of the Concept Mastery Routine. 

2. Briefly describe each phase of the Cue-Do-Review Sequence. 

3. List and explain the Linking Steps of the Concept Mastery Routine. 

4. List the three ways characteristics of a concept are classified in the Concept Mastery 
Routine. 

5. List the elements present in a good definition for a concept. 

6. Describe criteria to be considered when selecting concepts to be taught using the 
Concept Mastery Routine. 

7. Describe what a teacher should do to prepare to use the Concept Mastery Routine 
with students. 




73 



78 



Evaluation Guidelines f or the Knowledge Test 



1. Briefly describe the purpose of the Concept Mastery Routine. (3 points 

possible) 

1 Point - teach/ master/ understand 
1 Point - concepts/ big ideas 

1 Point - diverse/ academically diverse/ heterogeneous learners 

2. Briefly describe each phase of the Cue-Do-Review Sequence. (9 points possible) 

a. Cue: 

1 Point - explain that a Concept Diagram or the Concept Mastery Routine is 
going to be used 

1 Point - explain how the diagram or routine will help students learn 
1 Point - explain how students are to participate while the routine is used 

b. Do: 

1 Point - follow the Linking Steps 
1 Point - complete Concept Diagram 
1 Point - interact with students 

c. Review: 

1 Point - ask students questions 

1 Point - procedural and factual questions 

1 Point - check/ confirm/ determine student understanding 

3. List and explain the Linking Steps of the Concept Mastery Routine. (14 points 
possible) 

a. Step 1 

1 Point - Convey Targeted Concept Name 
1 Point - name the concept to be learned 

b. Step 2 
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1 Point - Offer Overall Concept Name 

1 Point - name a larger/ broader concept which subsumes targeted concept 

c. Step 3 

1 Point - Note Kev Words 

1 Point - brainstorm/ elicit words related to targeted concept 

d. Step 4 

1 Point - Classify Characteristics 

1 Point - identify always, sometimes, and never characteristics 

e. Step 5 

1 Point - Explore Examples 
1 Point - identify examples and nonexamples 

f. Step 6 

1 Point - Practice with Examples 
1 Point - practice with a new example/ nonexample 

g. Step 7 

1 Point - Tie Down a Definition 
1 Point - define the concept 

4. List the three ways characteristics of a concept are classified in the Concept 
Mastery Routine. (3 points possible) 

1 Point - always characteristics 

1 Point - sometimes characteristics 

1 Point - never characteristics 

5. List the elements present in a good definition for a concept. (3 points possible) 

1 Point - contain the targeted concept name 
1 Point - contain the overall concept name 
1 Point - contain the always characteristics 



O 

ERIC 



75 



80 



6. Describe criteria to be considered when selecting concepts to be taught using 
the Concept Mastery Routine. (3 points possible) 

1 Point - important for future learning/ foundational/ assumptions 

1 Point - abstractness of concept 

1 Point - relevance of concept 

1 Point - student's interest in concept 

1 Point - complexity of concept 

7. Describe what a teacher should do to prepare to use the Concept Mastery 
Routine with students. (3 points possible) 

1 Point - teacher must select concept to be taught using routine 

1 Point - students must have prior knowledge of the concept 

1 Point - teacher must prepare a draft concept diagram 
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Diagram Test Number: 

The Concept Mastery Routine 

University of Kansas 
Center for Research on Learning 



Name: 



Date: 



Directions: In the 10-minute time limit, complete the Concept Diagram for the 
targeted concept, automobile. Please note, only the content of your responces will be 
scored (not spelling,grammar, punctuation, etc.) 



Concept Diagram 



( ) c ) 

( ) ( ) 

( ) ( ) 
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Evaluation Guidelines for the Diagram Test 



1. Evaluate Targeted Concept Name: 5 points possible 

The Targeted Concept Name is a commonly accepted generic term for a class 
or category into which events, ideas, or objects can be grouped. 

a. Score 5 points if written in the rectangle 1 is a relevant word or phrase 
naming a the targeted concept - automobile. 

b. Score 0 points if written in rectangle 1 is not a meaningful word or phrase 
naming the targeted concept- automobile. 



2. Evaluate Overall Concept Name: 5 points possible 

The Overall Concept Name specifies a category or class into which the 
Targeted Concept Name and similar concepts can be grouped. 

a. Score 5 points if written in rectangle 2 is a relevant word or phrase naming 
an overall concept for automobile. 

Examples: Modes of Transportation 

Inventions 
Motorized Vehicles 

b. Score 0 points if written in rectangle 2 is not a relevant word or phrase 
naming an overall concept for automobile. 

Example: Motorcycles 



3. Note Key Words: 5 points possible 

Key words are words or phrases associated with the Targeted Concept Name. 
The Key Words for a Targeted Concept can be characteristics, examples, 
nonexamples, or other related concepts. 

a. Score 1 point for each word or phrase associated with automobile written in 
rectangle 3 of the concept diagram. 

Examples: Model T 

engine 

Boeing 747 (nonexample) 

b. Score 0 points for each non-meaningful word or phrase written in rectangle 
3 of the concept diagram. 

Examples: diagnostic questions 
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4. Evaluate Always Characteristics: 15 points possible 



Always Characteristics are qualities that all examples of the concept 
automobile must possess. 

a. Score 5 (up to a total of 15) points for each of the first three always 
characteristics written on the solid lines of the concept diagram. 

Examples: have a motor 

have tires 

b. Score 0 points for each inappropriate always characteristic. 

Examples: have 4 doors 

are green 

5. Evaluate Sometimes Characteristics: 15 points possible 

Sometimes Characteristics are qualities which may be present in some but not 
all examples of the concept automobile. 

a. Score 5 (up to a total of 15) points for each of the first three sometimes 
characteristics written on the wavy lines of the concept diagram. 

Examples: have a hatchback 

have air conditioning 

b. Score 0 points for each non-appropriate sometimes characteristic. 

Example: have motors 

6. Evaluate Never Characteristics: 15 points possible 

Never Characteristics are qualities not found in any examples of the concept 
automobile. 

a. Score 5 (up to a total of 15) points for each of the first three never 
characteristics written on the broken lines. 

Examples: have wings 

have only 2 wheels 

b. Score 0 points for each non-appropriate Never Characteristic. 

7. Evaluate Examples 

Examples are instances that posses all of the Always Characteristics and none 
of the Never Characteristics. 
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a. Score 5 (up to a total of 15) points for each of the first three examples 
written in the solid ovals if each possesses all of the Always and none of the 
Never Characteristics. 

Examples: truck 

Nash Rambler 

b. Score 0 points for each example which is an invalid illustration of the 
concept because it lacks one of the always present or possesses one of the 
never present characteristics. 

Examples: horse 



8. Evaluate Nonexamples 

Nonexamples are instances that possess one or more of the Never 
Characteristics or lack one or more of the Always Characteristics. 

a. Score 5 (up to a total of 15) points for each of the first three nonexamples 
written in the broken ovals if each lacks one or more Always Characteristics 
or possesses one or more Never Characteristics. 

Examples: bicycle 

Sherman Tank 

b. Score 0 points for a non-example which does not lack one or more of the 
Always or Never Characteristics. 

Examples: Model T Ford 

'57 Chevy 

9. Examine New Example: Maximum of 5 points 

Practicing with a new example is an instance in which either an example or 
nonexample for the Targeted Concept is written in the testing ground of the 
concept diagram. 

a. Score 5 (up to a total of 5) points if an example or nonexample of the 
Targeted Concept is written in the testing ground. 

Examples: lawn tractor 

Jeep CJ-6 

b. Score 0 points if a nonmeaningful example or nonexample for the Targeted 
Concept is written in the testing ground. 
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10. Examine Deflnition: Maximum of 15 points 

The definition for the Targeted Concept must be a complete sentence that 
includes the name of the Targeted Concept, the name of the Overall Concept, 
and all of the Always Characteristics. Moreover, it must be written for 
students to see. 

a. Score 5 points if the definition written in rectangle 7 includes the Targeted 
Concept name . 

Example: Automobile 

b. Score 5 points if the definition written in rectangle 7 includes the Overall 
Concept name. 

Example: Mode of Transportation 



c. Score 5 points if the definition written in rectangle 7 includes aU of the 
Always Characteristics. 

Example: has a motor 

has tires 

carries passengers 
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Implementation Checklist 

Concept Mastery Routine 



Date: Teacher: Class: Observer: _ 

CUE: 

Names process to be used to learn information 

Explains how process will help students learn 

Specifies what students need to do to participate in the process 

DO: 




I? 

I? 

I? 

I? 
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Evaluation Guidelines for the Implementation Checklist 



A. Cue 



1. Provide Advance Organizer: 15 points possible 

An Advance Organizer is a statement naming the process to be used to 
learn information, explaining how the process will help students learn, 
and specifying what students need to do to participate in the process. 

a. Score 5 points if the teacher names a specific process (i.e.. Concept 
Mastery Routine, Concept Diagram, Cooperative Learning) will be 
used to teacher a concept. 

b. Score 5 points if the teacher explains how the process will improve 
students learning of a concept. 

c. Score 5 points if the teacher specifies the students role in the 
process. 

c. Examples: 

- Teacher says, "Today I am going to teach using a Concept Diagram. 
This process will help clarify your understanding of a concept we have 
been studying. You are to participate in the lesson by sharing what you 
know about the concept and taking note on the concept diagram." 

- Teacher says, "In order to help you better understand the concept of a 
rhombus, I will review the past week's notes using the Concept 
Mastery Routine. While reviewing, you will need to share what you 
remember about rhombi and help place that information into the 
Concept Diagram." 



B. Dj2 

1. Convey the Targeted Concept Name: 5 points possible 

The Targeted Concept Name is a commonly accepted generic term for 
a class or category into which events, ideas, or objects can be grouped. 

a. Score 5 points if the concept is appropriate and specifies a 
commonly accepted generic term for a class or category into which 
events, ideas, or objects can be grouped. The Targeted Concept Name 
must be written on the chalkboard or overhead for students to see. 

b. Examples: 

- Teacher says, "Today we are going to study the rhombus," and writes 
the word "rhombus" on the board. 
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- Teacher says, "Let's consider amphibians," and writes the word 
"amphibians" on the overhead. 

2. Offer the Overall Concept Name: 5 points possible 

The Overall Concept Name specifies a category or class into which the 
Targeted Concept Name and similar concepts can be grouped. 

a. Score 5 points if the overall concept is appropriate and specifies a 
category or class into which the Targeted Concept Name and similar 
concepts can be grouped. The Overall Concept Name must be written 
on the board or overhead for students to see. 

b. Examples: 

- Teacher says, "The rhombus is a type of geometric figure," and 
writes the word "geometric figure" on the board. 

- Teacher says, "Amphibians are a type of vertebrate," and writes 
"venebrate" on the board. 

3. Note Key Words: Maximum of 5 points possible 

Key words are words or phrases students associate with the Targeted 
Concept Name. The Key Words for a Targeted Concept can be 
characteristics, examples, nonexamples, or other related concepts. 

a. Score 1 point for each meaningful word or phrase shared by a 
students and written on the board associated with the Targeted 
Concept. 

b. Examples: 

- For the Targeted Concept amphibians, a student says, "frogs," and the 
teacher writes "frogs" on the board. 

- For the Targeted Concept of rhombus, a students says, "four sides," 
and the teacher writes four sides on the board. 

4. Classify the Always, Sometimes, and Never Characteristics: 

Maximum of 45 points. 

Always Characteristics are qualities that all examples of the concept 
must possess. Sometimes Characteristics are qualities which may be 
present in some but not all examples of the concept. Never 
Characteristics are qualities not found in any examples of the concept. 

a. Score 5 (up to a total of 15) points for each appropriate characteristic 
written on the board labeled as always present. 

b. Examples: 
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- Always Characteristic for a rhombus - Teacher say, "Yes, has four 
sides is an Always characteristic," and writes "has four sides" on the 
board with an identification of being an Always Characteristic." 

- Always Characteristic for an amphibian - Teacher says, "Yes, are 
cold-blooded is an Always Characteristic of an amphibian," and on 
writes "are cold-blooded" on the board with an identification of it 
being an Always Characteristic.. 

c. Score 5 (up to a total of 15) points for each appropriate characteristic 
written on the board and labeled as sometimes present. 

d. Examples: 

- Sometimes Characteristic for a rhombus - Teacher says, "can have 
four equal angles is a Sometimes Characteristic of rhombi," and writes 
"four equal angles" on the board noting it is a Sometimes 
Characteristic.. 

- Sometimes Characteristic for a amphibian - Teacher says, "have 4 
legs is another Sometimes Characteristic," and writes "have 4 legs" on 
the board identifying it as a Sometimes Characteristic. 

e. Score 5 (up to a total of 15) points for each appropriate characteristic 
written on the board and labeled as never present. 

f. Examples: 

- Never Characteristic for a rhombus - Student says, "two unparallel 
sides is a Never Characteristic," and the teacher writes "2 unparallel 
sides" on the board noting it as a Never Characteristic. 

- Never Characteristic for an amphibian - Teachers says, "warm- 
blooded, " and on writes it on the board under the header Never 
Characteristic. 

5. Explore Examples and Nonexamples: Maximum of 30 points. 

Examples are instances that posses all of the Always Characteristics 
and none of the Never Characteristics. Nonexamples are instances that 
possess one or more of the Never Characteristics or lack one or more 
of the Always Characteristics. 

a. Score 5 (up to a total of 15) points for each example if the example 
possesses all of the Always, none of the Never Characteristics, and is 
written on the board. 

b. Examples: 

- Example of a rhombus - a square. 
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- Example of an amphibian - a frog. 



c. Score 5 (up to a total of 15) points for a nonexample if the 
nonexample lacks one or more Always Characteristics, or possesses 
one or more Never Characteristics, and is written on the board. 

d. Examples: 

- Nonexample of a rhombus - a triangle. 

- Nonexample of an amphibian - a bird. 

6. Practice with a New Example: Maximum of 5 points 

Practicing with a new example is an instance in which the teacher tells 
students the name of either an example or nonexample for the Targeted 
Concept and provides students (without teacher support) the 
opportunity determine which of the two it is. 

a. Score 5 (up to a total of 5) points if the teacher provides students an 
example or nonexample of the Targeted Concept, time to determine 
which of the two it is, and writes the example or nonexample on the 
board. 

b. Examples: 

- Teacher says, "Students, determine for yourself whether a snake is an 
example of an amphibian or not." 

- Teacher says, "I am going to test your understanding of the concept. 

Is a diamond an example or nonexample of a rhombus? Be ready to 
tell me in two minutes!" 

7. Explore and Practice Example Links: Maximum of 5 points 

Links are when the teacher, using either an example or nonexample 
(during either the Explore Examples or Practice with New Examples 
step), says or prompts students to say whether an example or 
nonexample does or does not possess an Always or a Never 
Characteristic of the Targeted Concept. 

a. Score 1 point for each link the teacher makes between a 
characteristic and an example or nonexample. 

b. Examples: 

- The teacher asks, "Is a square a rhombi?", and a student replies, "A 
square has four sides and that is an always characteristic for rhombi." 
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- The teacher says, "A bird is warm-blooded, this is a never 
characteristic for amphibians." 

c. Score 1 additional point for each link the teacher highlights using a 
symbol such as a + or an 0, a star, or an underline. 

d. Examples: 

- For the example "square", the teachers places a + next to the always 
characteristic - has four sides, (has four sides +1 

- For the nonexample "bird", the teacher places a 0 next to the always 
characteristic for amphibians - is cold-blooded.( is cold-blooded O’) 

8. Tie Down a Definition: Maximum of 15 points 

The definition for the Targeted Concept must be a complete sentence 
that includes the name of the Targeted Concept, the name of the 
Overall Concept, and all of the Always Characteristics. Moreover, it 
must be written for students to see. 

a. Score 5 points if the Targeted Concept name is included in the 
definition. 

b. Score 5 points if the Overall Concept name is included in the 
definition. 

c. Score 5 points if all of the Always Characteristics are included in the 
definition. 



C. Review 

1. Checks Student Understanding: 5 points possible. 

The review comes at the end of a lesson. During the review, the 
teacher checks students understanding by asking questions about either 
what was learned or how it was learned. 

a. Score 1 point (up to a total of 5 points) for each question the teacher 
asks students about what was learned or how it was learned. 

b. Examples: 

- Teacher says, "Today we learned about what?" 

- Teacher says, "What is another example of the concept we did not 
discuss today?" 
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- Teacher says, "In what way do you suppose organizing information 
in a graphic format helps us understand this idea better?" 

^ Teachers says, "If you were teaching a group of younger students this 
idea, would you relate it to a bigger idea the students may be familiar 
with? Why?" 

D. Interactions 

An interaction is observed each time the teacher obtains a piece of verbal 
information about the targeted concept for the concept diagram or for the 
review. 

a. Score 1 point (up to a total of 20 points) each time the teacher interacts with 
student in obtaining information related to the concept. 

b. Examples; 

- Teacher asks, "Is this characteristic Always, Sometimes, or Never present for 
this concept?" and a students provides an answer. 

- Teacher asks, "What would be a good definition?" and a student provides 
one. 




88 9 3 



Implementation Recording Form 

Concept Mastery Routine 



Name 


ID# 


Date of 
Training? 


Anticipated 
date of 

implementation? 


Actual date of 
implementation? 


Number of 
times used? 
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Satisfaction Questionnaire 

Concept Mastery Routine 



Name: Date: 

Please rate today's program by circling the appropriate number. 

Disagree Agree 



1. 


I believe that I will remember everything covered 
today. 


1 


2 


3 


4 


5 


6 


7 


2. 


The training session kept me focused on the 
content throughout 


1 


2 


3 


4 


5 


6 


7 




















3. 


I am very confident that I will soon use the 
routine learned today. 


1 


2 


3 


4 


5 


6 


7 




















4. 


This training session made me very enthusiastic 
about the content covered. 


1 


2 


3 


4 


5 


6 


7 


5. 


It will be very easy to summarize for other what 
this routine is all about 


1 


2 


3 


4 


5 


6 


7 












<. 


... 




.'.v 


6. 


It was easy to concentrate on the 
content of this presentation. 


1 


2 


3 


4 


5 


6 


7 














V..-. 






7. 


I plan to implement the routine very soon. 


1 


2 


3 


4 


5 


6 


7 














.^. .Vi 




.X 


8. 


I had a lot of fun during this presentation. 


1 


2 


3 


4 


5 


6 


7 




















9. 


I clearly understand everything that was presented 
today. 


1 


2 


3 


4 


5 


6 


7 




















10. 


The training session was engaging throughout. 


1 


2 


3 


4 


5 


6 


7 




















11. 


I am looking forward to incorporating this routine 
into the teaching that I am already doing 


1 


2 


3 


4 


5 


6 


7 




. 
















12. 


This session was very enjoyable for me. 


1 


2 


3 


4 


5 


6 


7 










. 










13. 


This training session was superior to other 
training sessions I have attended. 


1 


2 


3 


4 


5 


6 


7 




















14. 


Overall, I was highly satisfied with this training 
session. 


1 


2 


3 


4 


5 


6 


7 



15. Please make additional comments about the training session on the back of this sheet. 
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Table 1 



Preservice Teachers’ Mean Percentage Scores for Pretest and P osttest Knowledge_and 
D'iapram Tests bv Treatment Group 



Knowledee Scores 



Treatment Group 


Pretest 


Posttest 


Pretest 


Posnest 


Virtual Workshop 










M 


.90% 


49.09% 


10.03% 


84.76% 


SD 


.67 


6.46 


7.47 


10.92 


Actual Workshop 










M 


2.30% 


53.99% 


8.30% 


84.14% 


SD 


.90 


6.31 


6.69 


11.44 


Table 2 










Inservice Teachers' Mean Percentage Scores for Pretest and Posttest Knowledee and 


Diagram Tests bv Treatment Group 










Knowledee Scores 


Diagram Scores 


Treatment Group 


Pretest 


Posnest 


Pretest 


Posnest 


Virtual Workshop 










M 


5.26% 


63.68% 


11.81% 


93.63% 




1.23 


4.15 


16.05 


10.95 


Actual Workshop 










M 


6.32% 


60.00% 


39.09% 


85.45% 


SD 


1.82 


3.56 


38.18 


7.42 
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CONCEPT DIAGRAM 
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EifiMre 2. Virtual Workshop Computer Screen Displaying Table of Contents 
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Figure 3. Preservice Teacher Satisfaction Questioiuiaire Results 
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Figure 4 . Inservice Teachers' #1 and #2 (Virtual Workshop Participants) 



/ 
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Figure 5 . Inservice Teachers' #3 and #4 (Virtual Workshop Participants) 



/ 
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Figure 6 . Inservice Teachers' #5 and #6 (Actual Workshop Participants) 
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Figure 7 . Inservice Teachers' #7 and #8 (Actual Workshop Participants) 






Days 



Figure 8. Inservice Teachers' Implementation Questionnaire Results 





100 



Virtual Workshop 
P^cipants 




101 



. 



\ 



o 

ERIC 



» o ^ o 

^ a u *3 

p 3 s 2 3 

p 2 G c g 

« rt “ Q « 

^ ^ u ^ U 

^ y o o. 

- g U ®- 

u 



oo c ^ w 

.S o 2 c 

5 1 1 i 8 ^ 

E 1 S5 « s 

S u c c ^ > 

— •= g " - S 

£• 




^ = § 



102 



Of!! 



cvj 



i 




as. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement (OERI) 
Educational Resources Information Center (ERIC) 




NOTICE 

REPRODUCTION BASTS 




This document is covered by a signed “Reproduction Release 
(Blanket)” form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release 
form (either “Specific Document” or “Blanket”). 



